{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Lb9q2_QZgdNk"
      },
      "source": [
        "<a target=\"_blank\" href=\"https://colab.research.google.com/github/AI4Finance-Foundation/FinRL-Tutorials/blob/master/2-Advance/FinRL_Ensemble_StockTrading_ICAIF_2020.ipynb\">\n",
        "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n",
        "</a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gXaoZs2lh1hi"
      },
      "source": [
        "# Deep Reinforcement Learning for Stock Trading from Scratch: Multiple Stock Trading Using Ensemble Strategy\n",
        "\n",
        "Tutorials to use OpenAI DRL to trade multiple stocks using ensemble strategy in one Jupyter Notebook | Presented at ICAIF 2020\n",
        "\n",
        "* This notebook is the reimplementation of our paper: Deep Reinforcement Learning for Automated Stock Trading: An Ensemble Strategy, using FinRL.\n",
        "* Check out medium blog for detailed explanations: https://medium.com/@ai4finance/deep-reinforcement-learning-for-automated-stock-trading-f1dad0126a02\n",
        "* Please report any issues to our Github: https://github.com/AI4Finance-LLC/FinRL-Library/issues\n",
        "* **Pytorch Version**\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lGunVt8oLCVS"
      },
      "source": [
        "# Content"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HOzAKQ-SLGX6"
      },
      "source": [
        "* [1. Problem Definition](#0)\n",
        "* [2. Getting Started - Load Python packages](#1)\n",
        "    * [2.1. Install Packages](#1.1)    \n",
        "    * [2.2. Check Additional Packages](#1.2)\n",
        "    * [2.3. Import Packages](#1.3)\n",
        "    * [2.4. Create Folders](#1.4)\n",
        "* [3. Download Data](#2)\n",
        "* [4. Preprocess Data](#3)        \n",
        "    * [4.1. Technical Indicators](#3.1)\n",
        "    * [4.2. Perform Feature Engineering](#3.2)\n",
        "* [5.Build Environment](#4)  \n",
        "    * [5.1. Training & Trade Data Split](#4.1)\n",
        "    * [5.2. User-defined Environment](#4.2)   \n",
        "    * [5.3. Initialize Environment](#4.3)    \n",
        "* [6.Implement DRL Algorithms](#5)  \n",
        "* [7.Backtesting Performance](#6)  \n",
        "    * [7.1. BackTestStats](#6.1)\n",
        "    * [7.2. BackTestPlot](#6.2)   \n",
        "    * [7.3. Baseline Stats](#6.3)   \n",
        "    * [7.3. Compare to Stock Market Index](#6.4)             "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sApkDlD9LIZv"
      },
      "source": [
        "<a id='0'></a>\n",
        "# Part 1. Problem Definition"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HjLD2TZSLKZ-"
      },
      "source": [
        "This problem is to design an automated trading solution for single stock trading. We model the stock trading process as a Markov Decision Process (MDP). We then formulate our trading goal as a maximization problem.\n",
        "\n",
        "The algorithm is trained using Deep Reinforcement Learning (DRL) algorithms and the components of the reinforcement learning environment are:\n",
        "\n",
        "\n",
        "* Action: The action space describes the allowed actions that the agent interacts with the\n",
        "environment. Normally, a ∈ A includes three actions: a ∈ {−1, 0, 1}, where −1, 0, 1 represent\n",
        "selling, holding, and buying one stock. Also, an action can be carried upon multiple shares. We use\n",
        "an action space {−k, ..., −1, 0, 1, ..., k}, where k denotes the number of shares. For example, \"Buy\n",
        "10 shares of AAPL\" or \"Sell 10 shares of AAPL\" are 10 or −10, respectively\n",
        "\n",
        "* Reward function: r(s, a, s′) is the incentive mechanism for an agent to learn a better action. The change of the portfolio value when action a is taken at state s and arriving at new state s',  i.e., r(s, a, s′) = v′ − v, where v′ and v represent the portfolio\n",
        "values at state s′ and s, respectively\n",
        "\n",
        "* State: The state space describes the observations that the agent receives from the environment. Just as a human trader needs to analyze various information before executing a trade, so\n",
        "our trading agent observes many different features to better learn in an interactive environment.\n",
        "\n",
        "* Environment: Dow 30 consituents\n",
        "\n",
        "\n",
        "The data of the single stock that we will be using for this case study is obtained from Yahoo Finance API. The data contains Open-High-Low-Close price and volume.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ffsre789LY08"
      },
      "source": [
        "<a id='1'></a>\n",
        "# Part 2. Getting Started- Load Python Packages"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Uy5_PTmOh1hj"
      },
      "source": [
        "<a id='1.1'></a>\n",
        "## 2.1. Install all the packages through FinRL library\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mPT0ipYE28wL",
        "outputId": "31d6a8ab-c43f-4558-fbc2-6b633ef7654b"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: wrds in /usr/local/lib/python3.10/site-packages (3.2.0)\n",
            "Requirement already satisfied: numpy<1.27,>=1.26 in /usr/local/lib/python3.10/site-packages (from wrds) (1.26.4)\n",
            "Requirement already satisfied: packaging<23.3 in /usr/local/lib/python3.10/site-packages (from wrds) (23.2)\n",
            "Requirement already satisfied: pandas<2.3,>=2.2 in /usr/local/lib/python3.10/site-packages (from wrds) (2.2.1)\n",
            "Requirement already satisfied: psycopg2-binary<2.10,>=2.9 in /usr/local/lib/python3.10/site-packages (from wrds) (2.9.9)\n",
            "Requirement already satisfied: scipy<1.13,>=1.12 in /usr/local/lib/python3.10/site-packages (from wrds) (1.12.0)\n",
            "Requirement already satisfied: sqlalchemy<2.1,>=2 in /usr/local/lib/python3.10/site-packages (from wrds) (2.0.29)\n",
            "Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/site-packages (from pandas<2.3,>=2.2->wrds) (2.9.0.post0)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/site-packages (from pandas<2.3,>=2.2->wrds) (2024.1)\n",
            "Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.10/site-packages (from pandas<2.3,>=2.2->wrds) (2024.1)\n",
            "Requirement already satisfied: typing-extensions>=4.6.0 in /usr/local/lib/python3.10/site-packages (from sqlalchemy<2.1,>=2->wrds) (4.11.0)\n",
            "Requirement already satisfied: greenlet!=0.4.17 in /usr/local/lib/python3.10/site-packages (from sqlalchemy<2.1,>=2->wrds) (3.0.3)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas<2.3,>=2.2->wrds) (1.16.0)\n",
            "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
            "\u001b[0mRequirement already satisfied: swig in /usr/local/lib/python3.10/site-packages (4.2.1)\n",
            "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
            "\u001b[0m\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
            "\u001b[0m✨🍰✨ Everything looks OK!\n",
            "Collecting git+https://github.com/AI4Finance-Foundation/FinRL.git\n",
            "  Cloning https://github.com/AI4Finance-Foundation/FinRL.git to /tmp/pip-req-build-xp3z2_tj\n",
            "  Running command git clone --filter=blob:none --quiet https://github.com/AI4Finance-Foundation/FinRL.git /tmp/pip-req-build-xp3z2_tj\n",
            "  Resolved https://github.com/AI4Finance-Foundation/FinRL.git to commit 7b2f30302e787e9276f52823c87b7c2ade4203cf\n",
            "  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "  Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n",
            "Collecting elegantrl@ git+https://github.com/AI4Finance-Foundation/ElegantRL.git#egg=elegantrl (from finrl==0.3.6)\n",
            "  Cloning https://github.com/AI4Finance-Foundation/ElegantRL.git to /tmp/pip-install-fh0kgi8h/elegantrl_9258375a89e84ebb973a8518f78e4a92\n",
            "  Running command git clone --filter=blob:none --quiet https://github.com/AI4Finance-Foundation/ElegantRL.git /tmp/pip-install-fh0kgi8h/elegantrl_9258375a89e84ebb973a8518f78e4a92\n",
            "  Resolved https://github.com/AI4Finance-Foundation/ElegantRL.git to commit 87cf325af4c0608f1e74941091a3bc277dfaf739\n",
            "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: alpaca-trade-api<4,>=3 in /usr/local/lib/python3.10/site-packages (from finrl==0.3.6) (3.2.0)\n",
            "Requirement already satisfied: ccxt<4,>=3 in /usr/local/lib/python3.10/site-packages (from finrl==0.3.6) (3.1.60)\n",
            "Requirement already satisfied: exchange-calendars<5,>=4 in /usr/local/lib/python3.10/site-packages (from finrl==0.3.6) (4.5.3)\n",
            "Requirement already satisfied: jqdatasdk<2,>=1 in /usr/local/lib/python3.10/site-packages (from finrl==0.3.6) (1.9.3)\n",
            "Requirement already satisfied: pyfolio<0.10,>=0.9 in /usr/local/lib/python3.10/site-packages (from finrl==0.3.6) (0.9.2)\n",
            "Requirement already satisfied: pyportfolioopt<2,>=1 in /usr/local/lib/python3.10/site-packages (from finrl==0.3.6) (1.5.5)\n",
            "Requirement already satisfied: ray<3,>=2 in /usr/local/lib/python3.10/site-packages (from ray[default,tune]<3,>=2->finrl==0.3.6) (2.10.0)\n",
            "Requirement already satisfied: scikit-learn<2,>=1 in /usr/local/lib/python3.10/site-packages (from finrl==0.3.6) (1.4.1.post1)\n",
            "Requirement already satisfied: stable-baselines3>=2.0.0a5 in /usr/local/lib/python3.10/site-packages (from stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (2.3.0)\n",
            "Requirement already satisfied: stockstats<0.6,>=0.5 in /usr/local/lib/python3.10/site-packages (from finrl==0.3.6) (0.5.4)\n",
            "Requirement already satisfied: wrds<4,>=3 in /usr/local/lib/python3.10/site-packages (from finrl==0.3.6) (3.2.0)\n",
            "Requirement already satisfied: yfinance<0.3,>=0.2 in /usr/local/lib/python3.10/site-packages (from finrl==0.3.6) (0.2.37)\n",
            "Requirement already satisfied: pandas>=0.18.1 in /usr/local/lib/python3.10/site-packages (from alpaca-trade-api<4,>=3->finrl==0.3.6) (2.2.1)\n",
            "Requirement already satisfied: numpy>=1.11.1 in /usr/local/lib/python3.10/site-packages (from alpaca-trade-api<4,>=3->finrl==0.3.6) (1.26.4)\n",
            "Requirement already satisfied: requests<3,>2 in /usr/local/lib/python3.10/site-packages (from alpaca-trade-api<4,>=3->finrl==0.3.6) (2.31.0)\n",
            "Requirement already satisfied: urllib3<2,>1.24 in /usr/local/lib/python3.10/site-packages (from alpaca-trade-api<4,>=3->finrl==0.3.6) (1.26.18)\n",
            "Requirement already satisfied: websocket-client<2,>=0.56.0 in /usr/local/lib/python3.10/site-packages (from alpaca-trade-api<4,>=3->finrl==0.3.6) (1.7.0)\n",
            "Requirement already satisfied: websockets<11,>=9.0 in /usr/local/lib/python3.10/site-packages (from alpaca-trade-api<4,>=3->finrl==0.3.6) (10.4)\n",
            "Requirement already satisfied: msgpack==1.0.3 in /usr/local/lib/python3.10/site-packages (from alpaca-trade-api<4,>=3->finrl==0.3.6) (1.0.3)\n",
            "Requirement already satisfied: aiohttp<4,>=3.8.3 in /usr/local/lib/python3.10/site-packages (from alpaca-trade-api<4,>=3->finrl==0.3.6) (3.9.3)\n",
            "Requirement already satisfied: PyYAML==6.0.1 in /usr/local/lib/python3.10/site-packages (from alpaca-trade-api<4,>=3->finrl==0.3.6) (6.0.1)\n",
            "Requirement already satisfied: deprecation==2.1.0 in /usr/local/lib/python3.10/site-packages (from alpaca-trade-api<4,>=3->finrl==0.3.6) (2.1.0)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.10/site-packages (from deprecation==2.1.0->alpaca-trade-api<4,>=3->finrl==0.3.6) (23.2)\n",
            "Requirement already satisfied: setuptools>=60.9.0 in /usr/local/lib/python3.10/site-packages (from ccxt<4,>=3->finrl==0.3.6) (68.2.2)\n",
            "Requirement already satisfied: certifi>=2018.1.18 in /usr/local/lib/python3.10/site-packages (from ccxt<4,>=3->finrl==0.3.6) (2023.11.17)\n",
            "Requirement already satisfied: cryptography>=2.6.1 in /usr/local/lib/python3.10/site-packages (from ccxt<4,>=3->finrl==0.3.6) (42.0.5)\n",
            "Requirement already satisfied: aiodns>=1.1.1 in /usr/local/lib/python3.10/site-packages (from ccxt<4,>=3->finrl==0.3.6) (3.2.0)\n",
            "Requirement already satisfied: yarl>=1.7.2 in /usr/local/lib/python3.10/site-packages (from ccxt<4,>=3->finrl==0.3.6) (1.9.4)\n",
            "Requirement already satisfied: pyluach in /usr/local/lib/python3.10/site-packages (from exchange-calendars<5,>=4->finrl==0.3.6) (2.2.0)\n",
            "Requirement already satisfied: toolz in /usr/local/lib/python3.10/site-packages (from exchange-calendars<5,>=4->finrl==0.3.6) (0.12.1)\n",
            "Requirement already satisfied: tzdata in /usr/local/lib/python3.10/site-packages (from exchange-calendars<5,>=4->finrl==0.3.6) (2024.1)\n",
            "Requirement already satisfied: korean-lunar-calendar in /usr/local/lib/python3.10/site-packages (from exchange-calendars<5,>=4->finrl==0.3.6) (0.3.1)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.10/site-packages (from jqdatasdk<2,>=1->finrl==0.3.6) (1.16.0)\n",
            "Requirement already satisfied: SQLAlchemy>=1.2.8 in /usr/local/lib/python3.10/site-packages (from jqdatasdk<2,>=1->finrl==0.3.6) (2.0.29)\n",
            "Requirement already satisfied: thriftpy2>=0.3.9 in /usr/local/lib/python3.10/site-packages (from jqdatasdk<2,>=1->finrl==0.3.6) (0.4.20)\n",
            "Requirement already satisfied: pymysql>=0.7.6 in /usr/local/lib/python3.10/site-packages (from jqdatasdk<2,>=1->finrl==0.3.6) (1.1.0)\n",
            "Requirement already satisfied: ipython>=3.2.3 in /usr/local/lib/python3.10/site-packages (from pyfolio<0.10,>=0.9->finrl==0.3.6) (8.23.0)\n",
            "Requirement already satisfied: matplotlib>=1.4.0 in /usr/local/lib/python3.10/site-packages (from pyfolio<0.10,>=0.9->finrl==0.3.6) (3.8.4)\n",
            "Requirement already satisfied: pytz>=2014.10 in /usr/local/lib/python3.10/site-packages (from pyfolio<0.10,>=0.9->finrl==0.3.6) (2024.1)\n",
            "Requirement already satisfied: scipy>=0.14.0 in /usr/local/lib/python3.10/site-packages (from pyfolio<0.10,>=0.9->finrl==0.3.6) (1.12.0)\n",
            "Requirement already satisfied: seaborn>=0.7.1 in /usr/local/lib/python3.10/site-packages (from pyfolio<0.10,>=0.9->finrl==0.3.6) (0.13.2)\n",
            "Requirement already satisfied: empyrical>=0.5.0 in /usr/local/lib/python3.10/site-packages (from pyfolio<0.10,>=0.9->finrl==0.3.6) (0.5.5)\n",
            "Requirement already satisfied: cvxpy<2.0.0,>=1.1.19 in /usr/local/lib/python3.10/site-packages (from pyportfolioopt<2,>=1->finrl==0.3.6) (1.4.2)\n",
            "Requirement already satisfied: click>=7.0 in /usr/local/lib/python3.10/site-packages (from ray<3,>=2->ray[default,tune]<3,>=2->finrl==0.3.6) (8.1.7)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/site-packages (from ray<3,>=2->ray[default,tune]<3,>=2->finrl==0.3.6) (3.13.3)\n",
            "Requirement already satisfied: jsonschema in /usr/local/lib/python3.10/site-packages (from ray<3,>=2->ray[default,tune]<3,>=2->finrl==0.3.6) (4.21.1)\n",
            "Requirement already satisfied: protobuf!=3.19.5,>=3.15.3 in /usr/local/lib/python3.10/site-packages (from ray<3,>=2->ray[default,tune]<3,>=2->finrl==0.3.6) (4.25.3)\n",
            "Requirement already satisfied: aiosignal in /usr/local/lib/python3.10/site-packages (from ray<3,>=2->ray[default,tune]<3,>=2->finrl==0.3.6) (1.3.1)\n",
            "Requirement already satisfied: frozenlist in /usr/local/lib/python3.10/site-packages (from ray<3,>=2->ray[default,tune]<3,>=2->finrl==0.3.6) (1.4.1)\n",
            "Requirement already satisfied: aiohttp-cors in /usr/local/lib/python3.10/site-packages (from ray[default,tune]<3,>=2->finrl==0.3.6) (0.7.0)\n",
            "Requirement already satisfied: colorful in /usr/local/lib/python3.10/site-packages (from ray[default,tune]<3,>=2->finrl==0.3.6) (0.5.6)\n",
            "Requirement already satisfied: py-spy>=0.2.0 in /usr/local/lib/python3.10/site-packages (from ray[default,tune]<3,>=2->finrl==0.3.6) (0.3.14)\n",
            "Requirement already satisfied: opencensus in /usr/local/lib/python3.10/site-packages (from ray[default,tune]<3,>=2->finrl==0.3.6) (0.11.4)\n",
            "Requirement already satisfied: pydantic!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,<3 in /usr/local/lib/python3.10/site-packages (from ray[default,tune]<3,>=2->finrl==0.3.6) (2.6.4)\n",
            "Requirement already satisfied: prometheus-client>=0.7.1 in /usr/local/lib/python3.10/site-packages (from ray[default,tune]<3,>=2->finrl==0.3.6) (0.20.0)\n",
            "Requirement already satisfied: smart-open in /usr/local/lib/python3.10/site-packages (from ray[default,tune]<3,>=2->finrl==0.3.6) (7.0.4)\n",
            "Requirement already satisfied: virtualenv!=20.21.1,>=20.0.24 in /usr/local/lib/python3.10/site-packages (from ray[default,tune]<3,>=2->finrl==0.3.6) (20.25.1)\n",
            "Requirement already satisfied: grpcio>=1.42.0 in /usr/local/lib/python3.10/site-packages (from ray[default,tune]<3,>=2->finrl==0.3.6) (1.62.1)\n",
            "Requirement already satisfied: tensorboardX>=1.9 in /usr/local/lib/python3.10/site-packages (from ray[default,tune]<3,>=2->finrl==0.3.6) (2.6.2.2)\n",
            "Requirement already satisfied: pyarrow>=6.0.1 in /usr/local/lib/python3.10/site-packages (from ray[default,tune]<3,>=2->finrl==0.3.6) (15.0.2)\n",
            "Requirement already satisfied: fsspec in /usr/local/lib/python3.10/site-packages (from ray[default,tune]<3,>=2->finrl==0.3.6) (2024.3.1)\n",
            "Requirement already satisfied: joblib>=1.2.0 in /usr/local/lib/python3.10/site-packages (from scikit-learn<2,>=1->finrl==0.3.6) (1.4.0)\n",
            "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/site-packages (from scikit-learn<2,>=1->finrl==0.3.6) (3.4.0)\n",
            "Requirement already satisfied: gymnasium<0.30,>=0.28.1 in /usr/local/lib/python3.10/site-packages (from stable-baselines3>=2.0.0a5->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (0.29.1)\n",
            "Requirement already satisfied: torch>=1.13 in /usr/local/lib/python3.10/site-packages (from stable-baselines3>=2.0.0a5->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (2.2.2)\n",
            "Requirement already satisfied: cloudpickle in /usr/local/lib/python3.10/site-packages (from stable-baselines3>=2.0.0a5->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (3.0.0)\n",
            "Requirement already satisfied: opencv-python in /usr/local/lib/python3.10/site-packages (from stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (4.9.0.80)\n",
            "Requirement already satisfied: pygame in /usr/local/lib/python3.10/site-packages (from stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (2.1.0)\n",
            "Requirement already satisfied: tensorboard>=2.9.1 in /usr/local/lib/python3.10/site-packages (from stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (2.16.2)\n",
            "Requirement already satisfied: psutil in /usr/local/lib/python3.10/site-packages (from stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (5.9.8)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.10/site-packages (from stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (4.66.1)\n",
            "Requirement already satisfied: rich in /usr/local/lib/python3.10/site-packages (from stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (13.7.1)\n",
            "Requirement already satisfied: shimmy~=1.3.0 in /usr/local/lib/python3.10/site-packages (from shimmy[atari]~=1.3.0; extra == \"extra\"->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (1.3.0)\n",
            "Requirement already satisfied: pillow in /usr/local/lib/python3.10/site-packages (from stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (10.3.0)\n",
            "Requirement already satisfied: autorom~=0.6.1 in /usr/local/lib/python3.10/site-packages (from autorom[accept-rom-license]~=0.6.1; extra == \"extra\"->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (0.6.1)\n",
            "Requirement already satisfied: psycopg2-binary<2.10,>=2.9 in /usr/local/lib/python3.10/site-packages (from wrds<4,>=3->finrl==0.3.6) (2.9.9)\n",
            "Requirement already satisfied: multitasking>=0.0.7 in /usr/local/lib/python3.10/site-packages (from yfinance<0.3,>=0.2->finrl==0.3.6) (0.0.11)\n",
            "Requirement already satisfied: lxml>=4.9.1 in /usr/local/lib/python3.10/site-packages (from yfinance<0.3,>=0.2->finrl==0.3.6) (5.2.1)\n",
            "Requirement already satisfied: appdirs>=1.4.4 in /usr/local/lib/python3.10/site-packages (from yfinance<0.3,>=0.2->finrl==0.3.6) (1.4.4)\n",
            "Requirement already satisfied: frozendict>=2.3.4 in /usr/local/lib/python3.10/site-packages (from yfinance<0.3,>=0.2->finrl==0.3.6) (2.4.1)\n",
            "Requirement already satisfied: peewee>=3.16.2 in /usr/local/lib/python3.10/site-packages (from yfinance<0.3,>=0.2->finrl==0.3.6) (3.17.1)\n",
            "Requirement already satisfied: beautifulsoup4>=4.11.1 in /usr/local/lib/python3.10/site-packages (from yfinance<0.3,>=0.2->finrl==0.3.6) (4.12.3)\n",
            "Requirement already satisfied: html5lib>=1.1 in /usr/local/lib/python3.10/site-packages (from yfinance<0.3,>=0.2->finrl==0.3.6) (1.1)\n",
            "Requirement already satisfied: gym in /usr/local/lib/python3.10/site-packages (from elegantrl@ git+https://github.com/AI4Finance-Foundation/ElegantRL.git#egg=elegantrl->finrl==0.3.6) (0.26.2)\n",
            "Requirement already satisfied: pycares>=4.0.0 in /usr/local/lib/python3.10/site-packages (from aiodns>=1.1.1->ccxt<4,>=3->finrl==0.3.6) (4.4.0)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/site-packages (from aiohttp<4,>=3.8.3->alpaca-trade-api<4,>=3->finrl==0.3.6) (23.2.0)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/site-packages (from aiohttp<4,>=3.8.3->alpaca-trade-api<4,>=3->finrl==0.3.6) (6.0.5)\n",
            "Requirement already satisfied: async-timeout<5.0,>=4.0 in /usr/local/lib/python3.10/site-packages (from aiohttp<4,>=3.8.3->alpaca-trade-api<4,>=3->finrl==0.3.6) (4.0.3)\n",
            "Requirement already satisfied: AutoROM.accept-rom-license in /usr/local/lib/python3.10/site-packages (from autorom[accept-rom-license]~=0.6.1; extra == \"extra\"->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (0.6.1)\n",
            "Requirement already satisfied: soupsieve>1.2 in /usr/local/lib/python3.10/site-packages (from beautifulsoup4>=4.11.1->yfinance<0.3,>=0.2->finrl==0.3.6) (2.5)\n",
            "Requirement already satisfied: cffi>=1.12 in /usr/local/lib/python3.10/site-packages (from cryptography>=2.6.1->ccxt<4,>=3->finrl==0.3.6) (1.16.0)\n",
            "Requirement already satisfied: osqp>=0.6.2 in /usr/local/lib/python3.10/site-packages (from cvxpy<2.0.0,>=1.1.19->pyportfolioopt<2,>=1->finrl==0.3.6) (0.6.4)\n",
            "Requirement already satisfied: ecos>=2 in /usr/local/lib/python3.10/site-packages (from cvxpy<2.0.0,>=1.1.19->pyportfolioopt<2,>=1->finrl==0.3.6) (2.0.13)\n",
            "Requirement already satisfied: clarabel>=0.5.0 in /usr/local/lib/python3.10/site-packages (from cvxpy<2.0.0,>=1.1.19->pyportfolioopt<2,>=1->finrl==0.3.6) (0.7.1)\n",
            "Requirement already satisfied: scs>=3.0 in /usr/local/lib/python3.10/site-packages (from cvxpy<2.0.0,>=1.1.19->pyportfolioopt<2,>=1->finrl==0.3.6) (3.2.4.post1)\n",
            "Requirement already satisfied: pybind11 in /usr/local/lib/python3.10/site-packages (from cvxpy<2.0.0,>=1.1.19->pyportfolioopt<2,>=1->finrl==0.3.6) (2.12.0)\n",
            "Requirement already satisfied: pandas-datareader>=0.2 in /usr/local/lib/python3.10/site-packages (from empyrical>=0.5.0->pyfolio<0.10,>=0.9->finrl==0.3.6) (0.10.0)\n",
            "Requirement already satisfied: typing-extensions>=4.3.0 in /usr/local/lib/python3.10/site-packages (from gymnasium<0.30,>=0.28.1->stable-baselines3>=2.0.0a5->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (4.11.0)\n",
            "Requirement already satisfied: farama-notifications>=0.0.1 in /usr/local/lib/python3.10/site-packages (from gymnasium<0.30,>=0.28.1->stable-baselines3>=2.0.0a5->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (0.0.4)\n",
            "Requirement already satisfied: webencodings in /usr/local/lib/python3.10/site-packages (from html5lib>=1.1->yfinance<0.3,>=0.2->finrl==0.3.6) (0.5.1)\n",
            "Requirement already satisfied: decorator in /usr/local/lib/python3.10/site-packages (from ipython>=3.2.3->pyfolio<0.10,>=0.9->finrl==0.3.6) (5.1.1)\n",
            "Requirement already satisfied: jedi>=0.16 in /usr/local/lib/python3.10/site-packages (from ipython>=3.2.3->pyfolio<0.10,>=0.9->finrl==0.3.6) (0.19.1)\n",
            "Requirement already satisfied: matplotlib-inline in /usr/local/lib/python3.10/site-packages (from ipython>=3.2.3->pyfolio<0.10,>=0.9->finrl==0.3.6) (0.1.6)\n",
            "Requirement already satisfied: prompt-toolkit<3.1.0,>=3.0.41 in /usr/local/lib/python3.10/site-packages (from ipython>=3.2.3->pyfolio<0.10,>=0.9->finrl==0.3.6) (3.0.43)\n",
            "Requirement already satisfied: pygments>=2.4.0 in /usr/local/lib/python3.10/site-packages (from ipython>=3.2.3->pyfolio<0.10,>=0.9->finrl==0.3.6) (2.17.2)\n",
            "Requirement already satisfied: stack-data in /usr/local/lib/python3.10/site-packages (from ipython>=3.2.3->pyfolio<0.10,>=0.9->finrl==0.3.6) (0.6.3)\n",
            "Requirement already satisfied: traitlets>=5.13.0 in /usr/local/lib/python3.10/site-packages (from ipython>=3.2.3->pyfolio<0.10,>=0.9->finrl==0.3.6) (5.14.2)\n",
            "Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/site-packages (from ipython>=3.2.3->pyfolio<0.10,>=0.9->finrl==0.3.6) (1.2.0)\n",
            "Requirement already satisfied: pexpect>4.3 in /usr/local/lib/python3.10/site-packages (from ipython>=3.2.3->pyfolio<0.10,>=0.9->finrl==0.3.6) (4.9.0)\n",
            "Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/site-packages (from matplotlib>=1.4.0->pyfolio<0.10,>=0.9->finrl==0.3.6) (1.2.1)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/site-packages (from matplotlib>=1.4.0->pyfolio<0.10,>=0.9->finrl==0.3.6) (0.12.1)\n",
            "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/site-packages (from matplotlib>=1.4.0->pyfolio<0.10,>=0.9->finrl==0.3.6) (4.51.0)\n",
            "Requirement already satisfied: kiwisolver>=1.3.1 in /usr/local/lib/python3.10/site-packages (from matplotlib>=1.4.0->pyfolio<0.10,>=0.9->finrl==0.3.6) (1.4.5)\n",
            "Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/site-packages (from matplotlib>=1.4.0->pyfolio<0.10,>=0.9->finrl==0.3.6) (3.1.2)\n",
            "Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.10/site-packages (from matplotlib>=1.4.0->pyfolio<0.10,>=0.9->finrl==0.3.6) (2.9.0.post0)\n",
            "Requirement already satisfied: annotated-types>=0.4.0 in /usr/local/lib/python3.10/site-packages (from pydantic!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,<3->ray[default,tune]<3,>=2->finrl==0.3.6) (0.6.0)\n",
            "Requirement already satisfied: pydantic-core==2.16.3 in /usr/local/lib/python3.10/site-packages (from pydantic!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,<3->ray[default,tune]<3,>=2->finrl==0.3.6) (2.16.3)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/site-packages (from requests<3,>2->alpaca-trade-api<4,>=3->finrl==0.3.6) (3.3.2)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/site-packages (from requests<3,>2->alpaca-trade-api<4,>=3->finrl==0.3.6) (3.6)\n",
            "Requirement already satisfied: ale-py~=0.8.1 in /usr/local/lib/python3.10/site-packages (from shimmy[atari]~=1.3.0; extra == \"extra\"->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (0.8.1)\n",
            "Requirement already satisfied: greenlet!=0.4.17 in /usr/local/lib/python3.10/site-packages (from SQLAlchemy>=1.2.8->jqdatasdk<2,>=1->finrl==0.3.6) (3.0.3)\n",
            "Requirement already satisfied: absl-py>=0.4 in /usr/local/lib/python3.10/site-packages (from tensorboard>=2.9.1->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (2.1.0)\n",
            "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.10/site-packages (from tensorboard>=2.9.1->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (3.6)\n",
            "Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /usr/local/lib/python3.10/site-packages (from tensorboard>=2.9.1->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (0.7.2)\n",
            "Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.10/site-packages (from tensorboard>=2.9.1->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (3.0.2)\n",
            "Requirement already satisfied: ply<4.0,>=3.4 in /usr/local/lib/python3.10/site-packages (from thriftpy2>=0.3.9->jqdatasdk<2,>=1->finrl==0.3.6) (3.11)\n",
            "Requirement already satisfied: sympy in /usr/local/lib/python3.10/site-packages (from torch>=1.13->stable-baselines3>=2.0.0a5->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (1.12)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/site-packages (from torch>=1.13->stable-baselines3>=2.0.0a5->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (3.3)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/site-packages (from torch>=1.13->stable-baselines3>=2.0.0a5->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (3.1.3)\n",
            "Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /usr/local/lib/python3.10/site-packages (from torch>=1.13->stable-baselines3>=2.0.0a5->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (12.1.105)\n",
            "Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /usr/local/lib/python3.10/site-packages (from torch>=1.13->stable-baselines3>=2.0.0a5->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (12.1.105)\n",
            "Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /usr/local/lib/python3.10/site-packages (from torch>=1.13->stable-baselines3>=2.0.0a5->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (12.1.105)\n",
            "Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /usr/local/lib/python3.10/site-packages (from torch>=1.13->stable-baselines3>=2.0.0a5->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (8.9.2.26)\n",
            "Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /usr/local/lib/python3.10/site-packages (from torch>=1.13->stable-baselines3>=2.0.0a5->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (12.1.3.1)\n",
            "Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /usr/local/lib/python3.10/site-packages (from torch>=1.13->stable-baselines3>=2.0.0a5->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (11.0.2.54)\n",
            "Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /usr/local/lib/python3.10/site-packages (from torch>=1.13->stable-baselines3>=2.0.0a5->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (10.3.2.106)\n",
            "Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /usr/local/lib/python3.10/site-packages (from torch>=1.13->stable-baselines3>=2.0.0a5->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (11.4.5.107)\n",
            "Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /usr/local/lib/python3.10/site-packages (from torch>=1.13->stable-baselines3>=2.0.0a5->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (12.1.0.106)\n",
            "Requirement already satisfied: nvidia-nccl-cu12==2.19.3 in /usr/local/lib/python3.10/site-packages (from torch>=1.13->stable-baselines3>=2.0.0a5->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (2.19.3)\n",
            "Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /usr/local/lib/python3.10/site-packages (from torch>=1.13->stable-baselines3>=2.0.0a5->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (12.1.105)\n",
            "Requirement already satisfied: triton==2.2.0 in /usr/local/lib/python3.10/site-packages (from torch>=1.13->stable-baselines3>=2.0.0a5->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (2.2.0)\n",
            "Requirement already satisfied: nvidia-nvjitlink-cu12 in /usr/local/lib/python3.10/site-packages (from nvidia-cusolver-cu12==11.4.5.107->torch>=1.13->stable-baselines3>=2.0.0a5->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (12.4.127)\n",
            "Requirement already satisfied: distlib<1,>=0.3.7 in /usr/local/lib/python3.10/site-packages (from virtualenv!=20.21.1,>=20.0.24->ray[default,tune]<3,>=2->finrl==0.3.6) (0.3.8)\n",
            "Requirement already satisfied: platformdirs<5,>=3.9.1 in /usr/local/lib/python3.10/site-packages (from virtualenv!=20.21.1,>=20.0.24->ray[default,tune]<3,>=2->finrl==0.3.6) (4.1.0)\n",
            "Requirement already satisfied: gym-notices>=0.0.4 in /usr/local/lib/python3.10/site-packages (from gym->elegantrl@ git+https://github.com/AI4Finance-Foundation/ElegantRL.git#egg=elegantrl->finrl==0.3.6) (0.0.8)\n",
            "Requirement already satisfied: box2d-py==2.3.5 in /usr/local/lib/python3.10/site-packages (from gym[box2d]->elegantrl@ git+https://github.com/AI4Finance-Foundation/ElegantRL.git#egg=elegantrl->finrl==0.3.6) (2.3.5)\n",
            "Requirement already satisfied: swig==4.* in /usr/local/lib/python3.10/site-packages (from gym[box2d]->elegantrl@ git+https://github.com/AI4Finance-Foundation/ElegantRL.git#egg=elegantrl->finrl==0.3.6) (4.2.1)\n",
            "Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /usr/local/lib/python3.10/site-packages (from jsonschema->ray<3,>=2->ray[default,tune]<3,>=2->finrl==0.3.6) (2023.12.1)\n",
            "Requirement already satisfied: referencing>=0.28.4 in /usr/local/lib/python3.10/site-packages (from jsonschema->ray<3,>=2->ray[default,tune]<3,>=2->finrl==0.3.6) (0.34.0)\n",
            "Requirement already satisfied: rpds-py>=0.7.1 in /usr/local/lib/python3.10/site-packages (from jsonschema->ray<3,>=2->ray[default,tune]<3,>=2->finrl==0.3.6) (0.18.0)\n",
            "Requirement already satisfied: opencensus-context>=0.1.3 in /usr/local/lib/python3.10/site-packages (from opencensus->ray[default,tune]<3,>=2->finrl==0.3.6) (0.1.3)\n",
            "Requirement already satisfied: google-api-core<3.0.0,>=1.0.0 in /usr/local/lib/python3.10/site-packages (from opencensus->ray[default,tune]<3,>=2->finrl==0.3.6) (2.18.0)\n",
            "Requirement already satisfied: markdown-it-py>=2.2.0 in /usr/local/lib/python3.10/site-packages (from rich->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (3.0.0)\n",
            "Requirement already satisfied: wrapt in /usr/local/lib/python3.10/site-packages (from smart-open->ray[default,tune]<3,>=2->finrl==0.3.6) (1.16.0)\n",
            "Requirement already satisfied: importlib-resources in /usr/local/lib/python3.10/site-packages (from ale-py~=0.8.1->shimmy[atari]~=1.3.0; extra == \"extra\"->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (6.4.0)\n",
            "Requirement already satisfied: pycparser in /usr/local/lib/python3.10/site-packages (from cffi>=1.12->cryptography>=2.6.1->ccxt<4,>=3->finrl==0.3.6) (2.21)\n",
            "Requirement already satisfied: googleapis-common-protos<2.0.dev0,>=1.56.2 in /usr/local/lib/python3.10/site-packages (from google-api-core<3.0.0,>=1.0.0->opencensus->ray[default,tune]<3,>=2->finrl==0.3.6) (1.63.0)\n",
            "Requirement already satisfied: proto-plus<2.0.0dev,>=1.22.3 in /usr/local/lib/python3.10/site-packages (from google-api-core<3.0.0,>=1.0.0->opencensus->ray[default,tune]<3,>=2->finrl==0.3.6) (1.23.0)\n",
            "Requirement already satisfied: google-auth<3.0.dev0,>=2.14.1 in /usr/local/lib/python3.10/site-packages (from google-api-core<3.0.0,>=1.0.0->opencensus->ray[default,tune]<3,>=2->finrl==0.3.6) (2.29.0)\n",
            "Requirement already satisfied: parso<0.9.0,>=0.8.3 in /usr/local/lib/python3.10/site-packages (from jedi>=0.16->ipython>=3.2.3->pyfolio<0.10,>=0.9->finrl==0.3.6) (0.8.4)\n",
            "Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.10/site-packages (from markdown-it-py>=2.2.0->rich->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (0.1.2)\n",
            "Requirement already satisfied: qdldl in /usr/local/lib/python3.10/site-packages (from osqp>=0.6.2->cvxpy<2.0.0,>=1.1.19->pyportfolioopt<2,>=1->finrl==0.3.6) (0.1.7.post1)\n",
            "Requirement already satisfied: ptyprocess>=0.5 in /usr/local/lib/python3.10/site-packages (from pexpect>4.3->ipython>=3.2.3->pyfolio<0.10,>=0.9->finrl==0.3.6) (0.7.0)\n",
            "Requirement already satisfied: wcwidth in /usr/local/lib/python3.10/site-packages (from prompt-toolkit<3.1.0,>=3.0.41->ipython>=3.2.3->pyfolio<0.10,>=0.9->finrl==0.3.6) (0.2.13)\n",
            "Requirement already satisfied: MarkupSafe>=2.1.1 in /usr/local/lib/python3.10/site-packages (from werkzeug>=1.0.1->tensorboard>=2.9.1->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (2.1.5)\n",
            "Requirement already satisfied: executing>=1.2.0 in /usr/local/lib/python3.10/site-packages (from stack-data->ipython>=3.2.3->pyfolio<0.10,>=0.9->finrl==0.3.6) (2.0.1)\n",
            "Requirement already satisfied: asttokens>=2.1.0 in /usr/local/lib/python3.10/site-packages (from stack-data->ipython>=3.2.3->pyfolio<0.10,>=0.9->finrl==0.3.6) (2.4.1)\n",
            "Requirement already satisfied: pure-eval in /usr/local/lib/python3.10/site-packages (from stack-data->ipython>=3.2.3->pyfolio<0.10,>=0.9->finrl==0.3.6) (0.2.2)\n",
            "Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/site-packages (from sympy->torch>=1.13->stable-baselines3>=2.0.0a5->stable-baselines3[extra]>=2.0.0a5->finrl==0.3.6) (1.3.0)\n",
            "Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.10/site-packages (from google-auth<3.0.dev0,>=2.14.1->google-api-core<3.0.0,>=1.0.0->opencensus->ray[default,tune]<3,>=2->finrl==0.3.6) (5.3.3)\n",
            "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.10/site-packages (from google-auth<3.0.dev0,>=2.14.1->google-api-core<3.0.0,>=1.0.0->opencensus->ray[default,tune]<3,>=2->finrl==0.3.6) (0.4.0)\n",
            "Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.10/site-packages (from google-auth<3.0.dev0,>=2.14.1->google-api-core<3.0.0,>=1.0.0->opencensus->ray[default,tune]<3,>=2->finrl==0.3.6) (4.9)\n",
            "Requirement already satisfied: pyasn1<0.7.0,>=0.4.6 in /usr/local/lib/python3.10/site-packages (from pyasn1-modules>=0.2.1->google-auth<3.0.dev0,>=2.14.1->google-api-core<3.0.0,>=1.0.0->opencensus->ray[default,tune]<3,>=2->finrl==0.3.6) (0.6.0)\n",
            "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
            "\u001b[0m"
          ]
        }
      ],
      "source": [
        "# ## install finrl library\n",
        "!pip install wrds\n",
        "!pip install swig\n",
        "!pip install -q condacolab\n",
        "import condacolab\n",
        "condacolab.install()\n",
        "!apt-get update -y -qq && apt-get install -y -qq cmake libopenmpi-dev python3-dev zlib1g-dev libgl1-mesa-glx swig\n",
        "!pip install git+https://github.com/AI4Finance-Foundation/FinRL.git\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "osBHhVysOEzi"
      },
      "source": [
        "\n",
        "<a id='1.2'></a>\n",
        "## 2.2. Check if the additional packages needed are present, if not install them.\n",
        "* Yahoo Finance API\n",
        "* pandas\n",
        "* numpy\n",
        "* matplotlib\n",
        "* stockstats\n",
        "* OpenAI gym\n",
        "* stable-baselines\n",
        "* tensorflow\n",
        "* pyfolio"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nGv01K8Sh1hn"
      },
      "source": [
        "<a id='1.3'></a>\n",
        "## 2.3. Import Packages"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "EeMK7Uentj1V"
      },
      "outputs": [],
      "source": [
        "import warnings\n",
        "warnings.filterwarnings(\"ignore\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "lPqeTTwoh1hn"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "import matplotlib\n",
        "import matplotlib.pyplot as plt\n",
        "# matplotlib.use('Agg')\n",
        "import datetime\n",
        "\n",
        "%matplotlib inline\n",
        "from finrl.config_tickers import DOW_30_TICKER\n",
        "from finrl.meta.preprocessor.yahoodownloader import YahooDownloader\n",
        "from finrl.meta.preprocessor.preprocessors import FeatureEngineer, data_split\n",
        "from finrl.meta.env_stock_trading.env_stocktrading import StockTradingEnv\n",
        "from finrl.agents.stablebaselines3.models import DRLAgent,DRLEnsembleAgent\n",
        "from finrl.plot import backtest_stats, backtest_plot, get_daily_return, get_baseline\n",
        "\n",
        "from pprint import pprint\n",
        "\n",
        "import sys\n",
        "sys.path.append(\"../FinRL-Library\")\n",
        "\n",
        "import itertools"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T2owTj985RW4"
      },
      "source": [
        "<a id='1.4'></a>\n",
        "## 2.4. Create Folders"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "w9A8CN5R5PuZ"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "from finrl.main import check_and_make_directories\n",
        "from finrl.config import (\n",
        "    DATA_SAVE_DIR,\n",
        "    TRAINED_MODEL_DIR,\n",
        "    TENSORBOARD_LOG_DIR,\n",
        "    RESULTS_DIR,\n",
        "    INDICATORS,\n",
        "    TRAIN_START_DATE,\n",
        "    TRAIN_END_DATE,\n",
        "    TEST_START_DATE,\n",
        "    TEST_END_DATE,\n",
        "    TRADE_START_DATE,\n",
        "    TRADE_END_DATE,\n",
        ")\n",
        "\n",
        "check_and_make_directories([DATA_SAVE_DIR, TRAINED_MODEL_DIR, TENSORBOARD_LOG_DIR, RESULTS_DIR])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A289rQWMh1hq"
      },
      "source": [
        "<a id='2'></a>\n",
        "# Part 3. Download Data\n",
        "Yahoo Finance is a website that provides stock data, financial news, financial reports, etc. All the data provided by Yahoo Finance is free.\n",
        "* FinRL uses a class **YahooDownloader** to fetch data from Yahoo Finance API\n",
        "* Call Limit: Using the Public API (without authentication), you are limited to 2,000 requests per hour per IP (or up to a total of 48,000 requests a day).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NPeQ7iS-LoMm"
      },
      "source": [
        "\n",
        "\n",
        "-----\n",
        "class YahooDownloader:\n",
        "    Provides methods for retrieving daily stock data from\n",
        "    Yahoo Finance API\n",
        "\n",
        "    Attributes\n",
        "    ----------\n",
        "        start_date : str\n",
        "            start date of the data (modified from config.py)\n",
        "        end_date : str\n",
        "            end date of the data (modified from config.py)\n",
        "        ticker_list : list\n",
        "            a list of stock tickers (modified from config.py)\n",
        "\n",
        "    Methods\n",
        "    -------\n",
        "    fetch_data()\n",
        "        Fetches data from yahoo API\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JzqRRTOX6aFu"
      },
      "outputs": [],
      "source": [
        "print(DOW_30_TICKER)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 18,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yCKm4om-s9kE",
        "outputId": "ee98004d-3f47-4daf-f73a-867e5faf268e"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "[*********************100%%**********************]  1 of 1 completed\n",
            "[*********************100%%**********************]  1 of 1 completed\n",
            "[*********************100%%**********************]  1 of 1 completed\n",
            "[*********************100%%**********************]  1 of 1 completed\n",
            "[*********************100%%**********************]  1 of 1 completed\n",
            "[*********************100%%**********************]  1 of 1 completed\n",
            "[*********************100%%**********************]  1 of 1 completed\n",
            "[*********************100%%**********************]  1 of 1 completed\n",
            "[*********************100%%**********************]  1 of 1 completed\n",
            "[*********************100%%**********************]  1 of 1 completed\n",
            "[*********************100%%**********************]  1 of 1 completed\n",
            "[*********************100%%**********************]  1 of 1 completed\n",
            "[*********************100%%**********************]  1 of 1 completed\n",
            "[*********************100%%**********************]  1 of 1 completed\n",
            "[*********************100%%**********************]  1 of 1 completed\n",
            "[*********************100%%**********************]  1 of 1 completed\n",
            "[*********************100%%**********************]  1 of 1 completed\n",
            "[*********************100%%**********************]  1 of 1 completed\n",
            "[*********************100%%**********************]  1 of 1 completed\n",
            "[*********************100%%**********************]  1 of 1 completed\n",
            "[*********************100%%**********************]  1 of 1 completed\n",
            "[*********************100%%**********************]  1 of 1 completed\n",
            "[*********************100%%**********************]  1 of 1 completed\n",
            "[*********************100%%**********************]  1 of 1 completed\n",
            "[*********************100%%**********************]  1 of 1 completed\n",
            "[*********************100%%**********************]  1 of 1 completed\n",
            "[*********************100%%**********************]  1 of 1 completed\n",
            "[*********************100%%**********************]  1 of 1 completed\n",
            "[*********************100%%**********************]  1 of 1 completed\n",
            "[*********************100%%**********************]  1 of 1 completed\n"
          ]
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Shape of DataFrame:  (97013, 8)\n"
          ]
        }
      ],
      "source": [
        "# TRAIN_START_DATE = '2009-04-01'\n",
        "# TRAIN_END_DATE = '2021-01-01'\n",
        "# TEST_START_DATE = '2021-01-01'\n",
        "# TEST_END_DATE = '2022-06-01'\n",
        "from finrl.meta.preprocessor.yahoodownloader import YahooDownloader\n",
        "from finrl.config_tickers import DOW_30_TICKER\n",
        "\n",
        "TRAIN_START_DATE = '2010-01-01'\n",
        "TRAIN_END_DATE = '2021-10-01'\n",
        "TEST_START_DATE = '2021-10-01'\n",
        "TEST_END_DATE = '2023-03-01'\n",
        "\n",
        "df = YahooDownloader(start_date = TRAIN_START_DATE,\n",
        "                     end_date = TEST_END_DATE,\n",
        "                     ticker_list = DOW_30_TICKER).fetch_data()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uqC6c40Zh1iH"
      },
      "source": [
        "# Part 4: Preprocess Data\n",
        "Data preprocessing is a crucial step for training a high quality machine learning model. We need to check for missing data and do feature engineering in order to convert the data into a model-ready state.\n",
        "* Add technical indicators. In practical trading, various information needs to be taken into account, for example the historical stock prices, current holding shares, technical indicators, etc. In this article, we demonstrate two trend-following technical indicators: MACD and RSI.\n",
        "* Add turbulence index. Risk-aversion reflects whether an investor will choose to preserve the capital. It also influences one's trading strategy when facing different market volatility level. To control the risk in a worst-case scenario, such as financial crisis of 2007–2008, FinRL employs the financial turbulence index that measures extreme asset price fluctuation."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 19,
      "metadata": {
        "id": "kM5bH9uroCeg"
      },
      "outputs": [],
      "source": [
        " INDICATORS = ['macd',\n",
        "               'rsi_30',\n",
        "               'cci_30',\n",
        "               'dx_30']"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jgXfBcjxtj1a",
        "outputId": "07518687-7a3a-43ee-d500-ac50095b512a"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Successfully added technical indicators\n",
            "Successfully added turbulence index\n"
          ]
        }
      ],
      "source": [
        "from finrl.meta.preprocessor.preprocessors import FeatureEngineer\n",
        "fe = FeatureEngineer(use_technical_indicator=True,\n",
        "                     tech_indicator_list = INDICATORS,\n",
        "                     use_turbulence=True,\n",
        "                     user_defined_feature = False)\n",
        "\n",
        "processed = fe.preprocess_data(df)\n",
        "processed = processed.copy()\n",
        "processed = processed.fillna(0)\n",
        "processed = processed.replace(np.inf,0)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "len(state), len(data)"
      ],
      "metadata": {
        "id": "tiFA3pRWPQMO",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "94bbe31e-af8d-4489-d23f-6803a990c3e8"
      },
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(7, 13)"
            ]
          },
          "metadata": {},
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-QsYaY0Dh1iw"
      },
      "source": [
        "<a id='4'></a>\n",
        "# Part 5. Design Environment\n",
        "Considering the stochastic and interactive nature of the automated stock trading tasks, a financial task is modeled as a **Markov Decision Process (MDP)** problem. The training process involves observing stock price change, taking an action and reward's calculation to have the agent adjusting its strategy accordingly. By interacting with the environment, the trading agent will derive a trading strategy with the maximized rewards as time proceeds.\n",
        "\n",
        "Our trading environments, based on OpenAI Gym framework, simulate live stock markets with real market data according to the principle of time-driven simulation.\n",
        "\n",
        "The action space describes the allowed actions that the agent interacts with the environment. Normally, action a includes three actions: {-1, 0, 1}, where -1, 0, 1 represent selling, holding, and buying one share. Also, an action can be carried upon multiple shares. We use an action space {-k,…,-1, 0, 1, …, k}, where k denotes the number of shares to buy and -k denotes the number of shares to sell. For example, \"Buy 10 shares of AAPL\" or \"Sell 10 shares of AAPL\" are 10 or -10, respectively. The continuous action space needs to be normalized to [-1, 1], since the policy is defined on a Gaussian distribution, which needs to be normalized and symmetric."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Q2zqII8rMIqn",
        "outputId": "47edb3e0-2d83-4063-b2e8-c1d988e61da0"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Stock Dimension: 29, State Space: 175\n"
          ]
        }
      ],
      "source": [
        "stock_dimension = len(processed.tic.unique())\n",
        "state_space = 1 + 2*stock_dimension + len(INDICATORS)*stock_dimension\n",
        "print(f\"Stock Dimension: {stock_dimension}, State Space: {state_space}\")\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "id": "AWyp84Ltto19"
      },
      "outputs": [],
      "source": [
        "env_kwargs = {\n",
        "    \"hmax\": 100,\n",
        "    \"initial_amount\": 1000000,\n",
        "    \"buy_cost_pct\": 0.001,\n",
        "    \"sell_cost_pct\": 0.001,\n",
        "    \"state_space\": state_space,\n",
        "    \"stock_dim\": stock_dimension,\n",
        "    \"tech_indicator_list\": INDICATORS,\n",
        "    \"action_space\": stock_dimension,\n",
        "    \"reward_scaling\": 1e-4,\n",
        "    \"print_verbosity\":5\n",
        "\n",
        "}\n",
        "\n",
        "# buy_cost_list = sell_cost_list = [0.001] * stock_dimension\n",
        "# num_stock_shares = [0] * stock_dimension\n",
        "# env_kwargs = {\n",
        "#     \"hmax\": 100,\n",
        "#     \"initial_amount\": 1000000,\n",
        "#     \"num_stock_shares\": num_stock_shares,\n",
        "#     \"buy_cost_pct\": buy_cost_list,\n",
        "#     \"sell_cost_pct\": sell_cost_list,\n",
        "#     \"state_space\": state_space,\n",
        "#     \"stock_dim\": stock_dimension,\n",
        "#     \"tech_indicator_list\": INDICATORS,\n",
        "#     \"action_space\": stock_dimension,\n",
        "#     \"reward_scaling\": 1e-4\n",
        "# }"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HMNR5nHjh1iz"
      },
      "source": [
        "<a id='5'></a>\n",
        "# Part 6: Implement DRL Algorithms\n",
        "* The implementation of the DRL algorithms are based on **OpenAI Baselines** and **Stable Baselines**. Stable Baselines is a fork of OpenAI Baselines, with a major structural refactoring, and code cleanups.\n",
        "* FinRL library includes fine-tuned standard DRL algorithms, such as DQN, DDPG,\n",
        "Multi-Agent DDPG, PPO, SAC, A2C and TD3. We also allow users to\n",
        "design their own DRL algorithms by adapting these DRL algorithms.\n",
        "\n",
        "* In this notebook, we are training and validating 3 agents (A2C, PPO, DDPG) using Rolling-window Ensemble Method ([reference code](https://github.com/AI4Finance-LLC/Deep-Reinforcement-Learning-for-Automated-Stock-Trading-Ensemble-Strategy-ICAIF-2020/blob/80415db8fa7b2179df6bd7e81ce4fe8dbf913806/model/models.py#L92))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 23,
      "metadata": {
        "id": "v-gthCxMtj1d"
      },
      "outputs": [],
      "source": [
        "rebalance_window = 63 # rebalance_window is the number of days to retrain the model\n",
        "validation_window = 63 # validation_window is the number of days to do validation and trading (e.g. if validation_window=63, then both validation and trading period will be 63 days)\n",
        "\n",
        "ensemble_agent = DRLEnsembleAgent(df=processed,\n",
        "                 train_period=(TRAIN_START_DATE,TRAIN_END_DATE),\n",
        "                 val_test_period=(TEST_START_DATE,TEST_END_DATE),\n",
        "                 rebalance_window=rebalance_window,\n",
        "                 validation_window=validation_window,\n",
        "                 **env_kwargs)\n",
        "# e_train_gym = StockTradingEnv(df = processed, **env_kwargs)\n",
        "# agent = DRLAgent(e_train_gym)\n",
        "# if_using_a2c = True\n",
        "# model_a2c = agent.get_model(\"a2c\")\n",
        "# # if if_using_a2c:\n",
        "# #   tmp_path = RESULTS_DIR + '/a2c'\n",
        "# #   new_logger_a2c = configure(tmp_path, [\"stdout\", \"csv\", \"tensorboard\"])\n",
        "# #   model_a2c.set_logger(new_logger_a2c)\n",
        "# trained_a2c = agent.train_model(model=model_a2c,\n",
        "#                              tb_log_name='a2c',\n",
        "#                              total_timesteps=50000)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "id": "KsfEHa_Etj1d",
        "scrolled": false
      },
      "outputs": [],
      "source": [
        "A2C_model_kwargs = {\n",
        "                    'n_steps': 5,\n",
        "                    'ent_coef': 0.005,\n",
        "                    'learning_rate': 0.0007\n",
        "                    }\n",
        "\n",
        "PPO_model_kwargs = {\n",
        "                    \"ent_coef\":0.01,\n",
        "                    \"n_steps\": 2048,\n",
        "                    \"learning_rate\": 0.00025,\n",
        "                    \"batch_size\": 128\n",
        "                    }\n",
        "\n",
        "DDPG_model_kwargs = {\n",
        "                      #\"action_noise\":\"ornstein_uhlenbeck\",\n",
        "                      \"buffer_size\": 10_000,\n",
        "                      \"learning_rate\": 0.0005,\n",
        "                      \"batch_size\": 64\n",
        "                    }\n",
        "\n",
        "SAC_model_kwargs = {\n",
        "    \"batch_size\": 64,\n",
        "    \"buffer_size\": 100000,\n",
        "    \"learning_rate\": 0.0001,\n",
        "    \"learning_starts\": 100,\n",
        "    \"ent_coef\": \"auto_0.1\",\n",
        "}\n",
        "\n",
        "TD3_model_kwargs = {\"batch_size\": 100, \"buffer_size\": 1000000, \"learning_rate\": 0.0001}\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "timesteps_dict = {'a2c' : 10_000,\n",
        "                 'ppo' : 10_000,\n",
        "                 'ddpg' : 10_000,\n",
        "                 'sac' : 10_000,\n",
        "                 'td3' : 10_000\n",
        "                 }"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_1lyCECstj1e",
        "scrolled": true,
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "056b50cd-f8e8-4192-edd9-f570587ed923"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "============Start Ensemble Strategy============\n",
            "============================================\n",
            "turbulence_threshold:  201.74162030011615\n",
            "======Model training from:  2010-01-01 to  2021-10-04\n",
            "======a2c Training========\n",
            "{'n_steps': 5, 'ent_coef': 0.005, 'learning_rate': 0.0007}\n",
            "Using cuda device\n",
            "Logging to tensorboard_log/a2c/a2c_126_1\n",
            "---------------------------------------\n",
            "| time/                 |             |\n",
            "|    fps                | 83          |\n",
            "|    iterations         | 100         |\n",
            "|    time_elapsed       | 5           |\n",
            "|    total_timesteps    | 500         |\n",
            "| train/                |             |\n",
            "|    entropy_loss       | -41.1       |\n",
            "|    explained_variance | -0.589      |\n",
            "|    learning_rate      | 0.0007      |\n",
            "|    n_updates          | 99          |\n",
            "|    policy_loss        | -62.2       |\n",
            "|    reward             | -0.13443886 |\n",
            "|    std                | 0.998       |\n",
            "|    value_loss         | 3.54        |\n",
            "---------------------------------------\n",
            "--------------------------------------\n",
            "| time/                 |            |\n",
            "|    fps                | 93         |\n",
            "|    iterations         | 200        |\n",
            "|    time_elapsed       | 10         |\n",
            "|    total_timesteps    | 1000       |\n",
            "| train/                |            |\n",
            "|    entropy_loss       | -41        |\n",
            "|    explained_variance | -0.3       |\n",
            "|    learning_rate      | 0.0007     |\n",
            "|    n_updates          | 199        |\n",
            "|    policy_loss        | -58.5      |\n",
            "|    reward             | 0.42441055 |\n",
            "|    std                | 0.996      |\n",
            "|    value_loss         | 6.11       |\n",
            "--------------------------------------\n",
            "--------------------------------------\n",
            "| time/                 |            |\n",
            "|    fps                | 92         |\n",
            "|    iterations         | 300        |\n",
            "|    time_elapsed       | 16         |\n",
            "|    total_timesteps    | 1500       |\n",
            "| train/                |            |\n",
            "|    entropy_loss       | -41.1      |\n",
            "|    explained_variance | 0          |\n",
            "|    learning_rate      | 0.0007     |\n",
            "|    n_updates          | 299        |\n",
            "|    policy_loss        | -27.8      |\n",
            "|    reward             | -3.0476444 |\n",
            "|    std                | 0.998      |\n",
            "|    value_loss         | 2.41       |\n",
            "--------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 95        |\n",
            "|    iterations         | 400       |\n",
            "|    time_elapsed       | 21        |\n",
            "|    total_timesteps    | 2000      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -41.1     |\n",
            "|    explained_variance | -1.19e-07 |\n",
            "|    learning_rate      | 0.0007    |\n",
            "|    n_updates          | 399       |\n",
            "|    policy_loss        | 24        |\n",
            "|    reward             | 1.1522169 |\n",
            "|    std                | 0.999     |\n",
            "|    value_loss         | 2.67      |\n",
            "-------------------------------------\n",
            "---------------------------------------\n",
            "| time/                 |             |\n",
            "|    fps                | 97          |\n",
            "|    iterations         | 500         |\n",
            "|    time_elapsed       | 25          |\n",
            "|    total_timesteps    | 2500        |\n",
            "| train/                |             |\n",
            "|    entropy_loss       | -41.1       |\n",
            "|    explained_variance | 1.19e-07    |\n",
            "|    learning_rate      | 0.0007      |\n",
            "|    n_updates          | 499         |\n",
            "|    policy_loss        | -9.18       |\n",
            "|    reward             | 0.008832613 |\n",
            "|    std                | 0.999       |\n",
            "|    value_loss         | 3.65        |\n",
            "---------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 96        |\n",
            "|    iterations         | 600       |\n",
            "|    time_elapsed       | 31        |\n",
            "|    total_timesteps    | 3000      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -41       |\n",
            "|    explained_variance | 1.19e-07  |\n",
            "|    learning_rate      | 0.0007    |\n",
            "|    n_updates          | 599       |\n",
            "|    policy_loss        | -0.0801   |\n",
            "|    reward             | 0.8033523 |\n",
            "|    std                | 0.997     |\n",
            "|    value_loss         | 0.0839    |\n",
            "-------------------------------------\n",
            "---------------------------------------\n",
            "| time/                 |             |\n",
            "|    fps                | 97          |\n",
            "|    iterations         | 700         |\n",
            "|    time_elapsed       | 35          |\n",
            "|    total_timesteps    | 3500        |\n",
            "| train/                |             |\n",
            "|    entropy_loss       | -41.1       |\n",
            "|    explained_variance | 0.0154      |\n",
            "|    learning_rate      | 0.0007      |\n",
            "|    n_updates          | 699         |\n",
            "|    policy_loss        | -10.3       |\n",
            "|    reward             | 0.056252044 |\n",
            "|    std                | 0.999       |\n",
            "|    value_loss         | 0.266       |\n",
            "---------------------------------------\n",
            "---------------------------------------\n",
            "| time/                 |             |\n",
            "|    fps                | 96          |\n",
            "|    iterations         | 800         |\n",
            "|    time_elapsed       | 41          |\n",
            "|    total_timesteps    | 4000        |\n",
            "| train/                |             |\n",
            "|    entropy_loss       | -41.2       |\n",
            "|    explained_variance | 0.0216      |\n",
            "|    learning_rate      | 0.0007      |\n",
            "|    n_updates          | 799         |\n",
            "|    policy_loss        | 102         |\n",
            "|    reward             | -0.17337318 |\n",
            "|    std                | 1           |\n",
            "|    value_loss         | 7.78        |\n",
            "---------------------------------------\n",
            "--------------------------------------\n",
            "| time/                 |            |\n",
            "|    fps                | 97         |\n",
            "|    iterations         | 900        |\n",
            "|    time_elapsed       | 46         |\n",
            "|    total_timesteps    | 4500       |\n",
            "| train/                |            |\n",
            "|    entropy_loss       | -41.1      |\n",
            "|    explained_variance | 1.19e-07   |\n",
            "|    learning_rate      | 0.0007     |\n",
            "|    n_updates          | 899        |\n",
            "|    policy_loss        | -150       |\n",
            "|    reward             | -0.7243548 |\n",
            "|    std                | 1          |\n",
            "|    value_loss         | 16.4       |\n",
            "--------------------------------------\n",
            "---------------------------------------\n",
            "| time/                 |             |\n",
            "|    fps                | 96          |\n",
            "|    iterations         | 1000        |\n",
            "|    time_elapsed       | 51          |\n",
            "|    total_timesteps    | 5000        |\n",
            "| train/                |             |\n",
            "|    entropy_loss       | -41.2       |\n",
            "|    explained_variance | -0.0375     |\n",
            "|    learning_rate      | 0.0007      |\n",
            "|    n_updates          | 999         |\n",
            "|    policy_loss        | -510        |\n",
            "|    reward             | -0.63908553 |\n",
            "|    std                | 1           |\n",
            "|    value_loss         | 240         |\n",
            "---------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 97        |\n",
            "|    iterations         | 1100      |\n",
            "|    time_elapsed       | 56        |\n",
            "|    total_timesteps    | 5500      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -41.2     |\n",
            "|    explained_variance | 0.00788   |\n",
            "|    learning_rate      | 0.0007    |\n",
            "|    n_updates          | 1099      |\n",
            "|    policy_loss        | -248      |\n",
            "|    reward             | 3.7405448 |\n",
            "|    std                | 1         |\n",
            "|    value_loss         | 48.9      |\n",
            "-------------------------------------\n",
            "--------------------------------------\n",
            "| time/                 |            |\n",
            "|    fps                | 97         |\n",
            "|    iterations         | 1200       |\n",
            "|    time_elapsed       | 61         |\n",
            "|    total_timesteps    | 6000       |\n",
            "| train/                |            |\n",
            "|    entropy_loss       | -41.1      |\n",
            "|    explained_variance | 0.346      |\n",
            "|    learning_rate      | 0.0007     |\n",
            "|    n_updates          | 1199       |\n",
            "|    policy_loss        | -16.7      |\n",
            "|    reward             | -0.7040105 |\n",
            "|    std                | 1          |\n",
            "|    value_loss         | 0.666      |\n",
            "--------------------------------------\n",
            "--------------------------------------\n",
            "| time/                 |            |\n",
            "|    fps                | 97         |\n",
            "|    iterations         | 1300       |\n",
            "|    time_elapsed       | 66         |\n",
            "|    total_timesteps    | 6500       |\n",
            "| train/                |            |\n",
            "|    entropy_loss       | -41.1      |\n",
            "|    explained_variance | -0.0607    |\n",
            "|    learning_rate      | 0.0007     |\n",
            "|    n_updates          | 1299       |\n",
            "|    policy_loss        | 21.4       |\n",
            "|    reward             | 0.14980054 |\n",
            "|    std                | 0.999      |\n",
            "|    value_loss         | 0.792      |\n",
            "--------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 97        |\n",
            "|    iterations         | 1400      |\n",
            "|    time_elapsed       | 71        |\n",
            "|    total_timesteps    | 7000      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -41.1     |\n",
            "|    explained_variance | 0.15      |\n",
            "|    learning_rate      | 0.0007    |\n",
            "|    n_updates          | 1399      |\n",
            "|    policy_loss        | 148       |\n",
            "|    reward             | -1.891962 |\n",
            "|    std                | 0.999     |\n",
            "|    value_loss         | 17.6      |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 97        |\n",
            "|    iterations         | 1500      |\n",
            "|    time_elapsed       | 77        |\n",
            "|    total_timesteps    | 7500      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -41.1     |\n",
            "|    explained_variance | -0.0336   |\n",
            "|    learning_rate      | 0.0007    |\n",
            "|    n_updates          | 1499      |\n",
            "|    policy_loss        | 91.2      |\n",
            "|    reward             | -2.299666 |\n",
            "|    std                | 1         |\n",
            "|    value_loss         | 5.62      |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 97        |\n",
            "|    iterations         | 1600      |\n",
            "|    time_elapsed       | 81        |\n",
            "|    total_timesteps    | 8000      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -41.1     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0007    |\n",
            "|    n_updates          | 1599      |\n",
            "|    policy_loss        | -19.3     |\n",
            "|    reward             | 2.7779486 |\n",
            "|    std                | 0.999     |\n",
            "|    value_loss         | 13.9      |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 97        |\n",
            "|    iterations         | 1700      |\n",
            "|    time_elapsed       | 86        |\n",
            "|    total_timesteps    | 8500      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -41.1     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0007    |\n",
            "|    n_updates          | 1699      |\n",
            "|    policy_loss        | 36.7      |\n",
            "|    reward             | 5.3634834 |\n",
            "|    std                | 0.998     |\n",
            "|    value_loss         | 29.5      |\n",
            "-------------------------------------\n",
            "--------------------------------------\n",
            "| time/                 |            |\n",
            "|    fps                | 97         |\n",
            "|    iterations         | 1800       |\n",
            "|    time_elapsed       | 91         |\n",
            "|    total_timesteps    | 9000       |\n",
            "| train/                |            |\n",
            "|    entropy_loss       | -41.1      |\n",
            "|    explained_variance | 0.0493     |\n",
            "|    learning_rate      | 0.0007     |\n",
            "|    n_updates          | 1799       |\n",
            "|    policy_loss        | -60.4      |\n",
            "|    reward             | 0.34657776 |\n",
            "|    std                | 0.999      |\n",
            "|    value_loss         | 4.15       |\n",
            "--------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 98        |\n",
            "|    iterations         | 1900      |\n",
            "|    time_elapsed       | 96        |\n",
            "|    total_timesteps    | 9500      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -41.1     |\n",
            "|    explained_variance | -0.229    |\n",
            "|    learning_rate      | 0.0007    |\n",
            "|    n_updates          | 1899      |\n",
            "|    policy_loss        | 29.5      |\n",
            "|    reward             | 1.0733021 |\n",
            "|    std                | 1         |\n",
            "|    value_loss         | 2.12      |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 97        |\n",
            "|    iterations         | 2000      |\n",
            "|    time_elapsed       | 102       |\n",
            "|    total_timesteps    | 10000     |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -41.2     |\n",
            "|    explained_variance | -1.19e-07 |\n",
            "|    learning_rate      | 0.0007    |\n",
            "|    n_updates          | 1999      |\n",
            "|    policy_loss        | 41.5      |\n",
            "|    reward             | 0.5164767 |\n",
            "|    std                | 1         |\n",
            "|    value_loss         | 1.16      |\n",
            "-------------------------------------\n",
            "======a2c Validation from:  2021-10-04 to  2022-01-03\n",
            "a2c Sharpe Ratio:  0.12016203130695303\n",
            "======ddpg Training========\n",
            "{'buffer_size': 10000, 'learning_rate': 0.0005, 'batch_size': 64}\n",
            "Using cuda device\n",
            "Logging to tensorboard_log/ddpg/ddpg_126_1\n",
            "day: 2957, episode: 5\n",
            "begin_total_asset: 1000000.00\n",
            "end_total_asset: 4002721.16\n",
            "total_reward: 3002721.16\n",
            "total_cost: 6360.02\n",
            "total_trades: 48784\n",
            "Sharpe: 0.828\n",
            "=================================\n",
            "======ddpg Validation from:  2021-10-04 to  2022-01-03\n",
            "ddpg Sharpe Ratio:  0.23149939361322536\n",
            "======td3 Training========\n",
            "{'batch_size': 100, 'buffer_size': 1000000, 'learning_rate': 0.0001}\n",
            "Using cuda device\n",
            "Logging to tensorboard_log/td3/td3_126_1\n",
            "day: 2957, episode: 10\n",
            "begin_total_asset: 1000000.00\n",
            "end_total_asset: 5945616.62\n",
            "total_reward: 4945616.62\n",
            "total_cost: 2786.02\n",
            "total_trades: 38732\n",
            "Sharpe: 0.922\n",
            "=================================\n",
            "======td3 Validation from:  2021-10-04 to  2022-01-03\n",
            "td3 Sharpe Ratio:  0.12034224444593176\n",
            "======sac Training========\n",
            "{'batch_size': 64, 'buffer_size': 100000, 'learning_rate': 0.0001, 'learning_starts': 100, 'ent_coef': 'auto_0.1'}\n",
            "Using cuda device\n",
            "Logging to tensorboard_log/sac/sac_126_1\n",
            "day: 2957, episode: 15\n",
            "begin_total_asset: 1000000.00\n",
            "end_total_asset: 4555559.85\n",
            "total_reward: 3555559.85\n",
            "total_cost: 238769.98\n",
            "total_trades: 66550\n",
            "Sharpe: 0.861\n",
            "=================================\n",
            "======sac Validation from:  2021-10-04 to  2022-01-03\n",
            "sac Sharpe Ratio:  0.08822857821789602\n",
            "======ppo Training========\n",
            "{'ent_coef': 0.01, 'n_steps': 2048, 'learning_rate': 0.00025, 'batch_size': 128}\n",
            "Using cuda device\n",
            "Logging to tensorboard_log/ppo/ppo_126_1\n",
            "----------------------------------\n",
            "| time/              |           |\n",
            "|    fps             | 108       |\n",
            "|    iterations      | 1         |\n",
            "|    time_elapsed    | 18        |\n",
            "|    total_timesteps | 2048      |\n",
            "| train/             |           |\n",
            "|    reward          | 1.2595656 |\n",
            "----------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 107         |\n",
            "|    iterations           | 2           |\n",
            "|    time_elapsed         | 38          |\n",
            "|    total_timesteps      | 4096        |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.016799435 |\n",
            "|    clip_fraction        | 0.204       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -41.2       |\n",
            "|    explained_variance   | -0.0259     |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 3.89        |\n",
            "|    n_updates            | 10          |\n",
            "|    policy_gradient_loss | -0.0262     |\n",
            "|    reward               | 1.0748519   |\n",
            "|    std                  | 1           |\n",
            "|    value_loss           | 9.71        |\n",
            "-----------------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 106         |\n",
            "|    iterations           | 3           |\n",
            "|    time_elapsed         | 57          |\n",
            "|    total_timesteps      | 6144        |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.014081008 |\n",
            "|    clip_fraction        | 0.135       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -41.3       |\n",
            "|    explained_variance   | 0.0159      |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 13.2        |\n",
            "|    n_updates            | 20          |\n",
            "|    policy_gradient_loss | -0.0211     |\n",
            "|    reward               | -0.25309741 |\n",
            "|    std                  | 1           |\n",
            "|    value_loss           | 41.8        |\n",
            "-----------------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 106         |\n",
            "|    iterations           | 4           |\n",
            "|    time_elapsed         | 77          |\n",
            "|    total_timesteps      | 8192        |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.014428517 |\n",
            "|    clip_fraction        | 0.148       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -41.3       |\n",
            "|    explained_variance   | -0.0206     |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 30.4        |\n",
            "|    n_updates            | 30          |\n",
            "|    policy_gradient_loss | -0.0176     |\n",
            "|    reward               | 3.6489613   |\n",
            "|    std                  | 1.01        |\n",
            "|    value_loss           | 63.9        |\n",
            "-----------------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 106         |\n",
            "|    iterations           | 5           |\n",
            "|    time_elapsed         | 95          |\n",
            "|    total_timesteps      | 10240       |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.018376704 |\n",
            "|    clip_fraction        | 0.215       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -41.4       |\n",
            "|    explained_variance   | 0.0272      |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 5.93        |\n",
            "|    n_updates            | 40          |\n",
            "|    policy_gradient_loss | -0.0251     |\n",
            "|    reward               | -0.09722769 |\n",
            "|    std                  | 1.01        |\n",
            "|    value_loss           | 14.7        |\n",
            "-----------------------------------------\n",
            "======ppo Validation from:  2021-10-04 to  2022-01-03\n",
            "ppo Sharpe Ratio:  0.30010210770044654\n",
            "======Best Model Retraining from:  2010-01-01 to  2022-01-03\n",
            "======Trading from:  2022-01-03 to  2022-04-04\n",
            "============================================\n",
            "turbulence_threshold:  201.74162030011615\n",
            "======Model training from:  2010-01-01 to  2022-01-03\n",
            "======a2c Training========\n",
            "{'n_steps': 5, 'ent_coef': 0.005, 'learning_rate': 0.0007}\n",
            "Using cuda device\n",
            "Logging to tensorboard_log/a2c/a2c_189_1\n",
            "----------------------------------------\n",
            "| time/                 |              |\n",
            "|    fps                | 90           |\n",
            "|    iterations         | 100          |\n",
            "|    time_elapsed       | 5            |\n",
            "|    total_timesteps    | 500          |\n",
            "| train/                |              |\n",
            "|    entropy_loss       | -41.1        |\n",
            "|    explained_variance | 0.213        |\n",
            "|    learning_rate      | 0.0007       |\n",
            "|    n_updates          | 99           |\n",
            "|    policy_loss        | -82.9        |\n",
            "|    reward             | -0.023693616 |\n",
            "|    std                | 0.998        |\n",
            "|    value_loss         | 4.94         |\n",
            "----------------------------------------\n",
            "---------------------------------------\n",
            "| time/                 |             |\n",
            "|    fps                | 97          |\n",
            "|    iterations         | 200         |\n",
            "|    time_elapsed       | 10          |\n",
            "|    total_timesteps    | 1000        |\n",
            "| train/                |             |\n",
            "|    entropy_loss       | -41.1       |\n",
            "|    explained_variance | 0           |\n",
            "|    learning_rate      | 0.0007      |\n",
            "|    n_updates          | 199         |\n",
            "|    policy_loss        | -64.3       |\n",
            "|    reward             | -0.38760617 |\n",
            "|    std                | 1           |\n",
            "|    value_loss         | 6.33        |\n",
            "---------------------------------------\n",
            "--------------------------------------\n",
            "| time/                 |            |\n",
            "|    fps                | 95         |\n",
            "|    iterations         | 300        |\n",
            "|    time_elapsed       | 15         |\n",
            "|    total_timesteps    | 1500       |\n",
            "| train/                |            |\n",
            "|    entropy_loss       | -41.2      |\n",
            "|    explained_variance | 0.0205     |\n",
            "|    learning_rate      | 0.0007     |\n",
            "|    n_updates          | 299        |\n",
            "|    policy_loss        | 54.3       |\n",
            "|    reward             | -4.1570683 |\n",
            "|    std                | 1          |\n",
            "|    value_loss         | 4.02       |\n",
            "--------------------------------------\n",
            "---------------------------------------\n",
            "| time/                 |             |\n",
            "|    fps                | 98          |\n",
            "|    iterations         | 400         |\n",
            "|    time_elapsed       | 20          |\n",
            "|    total_timesteps    | 2000        |\n",
            "| train/                |             |\n",
            "|    entropy_loss       | -41.1       |\n",
            "|    explained_variance | 0.00152     |\n",
            "|    learning_rate      | 0.0007      |\n",
            "|    n_updates          | 399         |\n",
            "|    policy_loss        | 91.3        |\n",
            "|    reward             | -0.37606463 |\n",
            "|    std                | 0.999       |\n",
            "|    value_loss         | 14.6        |\n",
            "---------------------------------------\n",
            "--------------------------------------\n",
            "| time/                 |            |\n",
            "|    fps                | 99         |\n",
            "|    iterations         | 500        |\n",
            "|    time_elapsed       | 25         |\n",
            "|    total_timesteps    | 2500       |\n",
            "| train/                |            |\n",
            "|    entropy_loss       | -41        |\n",
            "|    explained_variance | 0          |\n",
            "|    learning_rate      | 0.0007     |\n",
            "|    n_updates          | 499        |\n",
            "|    policy_loss        | -72.9      |\n",
            "|    reward             | -1.6318904 |\n",
            "|    std                | 0.996      |\n",
            "|    value_loss         | 4.7        |\n",
            "--------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 98        |\n",
            "|    iterations         | 600       |\n",
            "|    time_elapsed       | 30        |\n",
            "|    total_timesteps    | 3000      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -41.1     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0007    |\n",
            "|    n_updates          | 599       |\n",
            "|    policy_loss        | 250       |\n",
            "|    reward             | 6.0129476 |\n",
            "|    std                | 0.997     |\n",
            "|    value_loss         | 57.9      |\n",
            "-------------------------------------\n",
            "---------------------------------------\n",
            "| time/                 |             |\n",
            "|    fps                | 99          |\n",
            "|    iterations         | 700         |\n",
            "|    time_elapsed       | 35          |\n",
            "|    total_timesteps    | 3500        |\n",
            "| train/                |             |\n",
            "|    entropy_loss       | -41         |\n",
            "|    explained_variance | -0.138      |\n",
            "|    learning_rate      | 0.0007      |\n",
            "|    n_updates          | 699         |\n",
            "|    policy_loss        | -199        |\n",
            "|    reward             | -0.07948906 |\n",
            "|    std                | 0.996       |\n",
            "|    value_loss         | 24.3        |\n",
            "---------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 98        |\n",
            "|    iterations         | 800       |\n",
            "|    time_elapsed       | 40        |\n",
            "|    total_timesteps    | 4000      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -41       |\n",
            "|    explained_variance | 0.00106   |\n",
            "|    learning_rate      | 0.0007    |\n",
            "|    n_updates          | 799       |\n",
            "|    policy_loss        | 35.2      |\n",
            "|    reward             | 1.1223269 |\n",
            "|    std                | 0.995     |\n",
            "|    value_loss         | 1.4       |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 98        |\n",
            "|    iterations         | 900       |\n",
            "|    time_elapsed       | 45        |\n",
            "|    total_timesteps    | 4500      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -41       |\n",
            "|    explained_variance | 0.104     |\n",
            "|    learning_rate      | 0.0007    |\n",
            "|    n_updates          | 899       |\n",
            "|    policy_loss        | -61.6     |\n",
            "|    reward             | 0.8920734 |\n",
            "|    std                | 0.995     |\n",
            "|    value_loss         | 4.01      |\n",
            "-------------------------------------\n",
            "---------------------------------------\n",
            "| time/                 |             |\n",
            "|    fps                | 98          |\n",
            "|    iterations         | 1000        |\n",
            "|    time_elapsed       | 50          |\n",
            "|    total_timesteps    | 5000        |\n",
            "| train/                |             |\n",
            "|    entropy_loss       | -41         |\n",
            "|    explained_variance | 0           |\n",
            "|    learning_rate      | 0.0007      |\n",
            "|    n_updates          | 999         |\n",
            "|    policy_loss        | 11.4        |\n",
            "|    reward             | -0.36836326 |\n",
            "|    std                | 0.993       |\n",
            "|    value_loss         | 1.83        |\n",
            "---------------------------------------\n",
            "------------------------------------\n",
            "| time/                 |          |\n",
            "|    fps                | 98       |\n",
            "|    iterations         | 1100     |\n",
            "|    time_elapsed       | 55       |\n",
            "|    total_timesteps    | 5500     |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -41      |\n",
            "|    explained_variance | 0        |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 1099     |\n",
            "|    policy_loss        | -33.7    |\n",
            "|    reward             | 3.212918 |\n",
            "|    std                | 0.994    |\n",
            "|    value_loss         | 3.55     |\n",
            "------------------------------------\n",
            "----------------------------------------\n",
            "| time/                 |              |\n",
            "|    fps                | 99           |\n",
            "|    iterations         | 1200         |\n",
            "|    time_elapsed       | 60           |\n",
            "|    total_timesteps    | 6000         |\n",
            "| train/                |              |\n",
            "|    entropy_loss       | -40.9        |\n",
            "|    explained_variance | 0            |\n",
            "|    learning_rate      | 0.0007       |\n",
            "|    n_updates          | 1199         |\n",
            "|    policy_loss        | -6.33        |\n",
            "|    reward             | -0.099947825 |\n",
            "|    std                | 0.993        |\n",
            "|    value_loss         | 2.43         |\n",
            "----------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 98        |\n",
            "|    iterations         | 1300      |\n",
            "|    time_elapsed       | 65        |\n",
            "|    total_timesteps    | 6500      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -41       |\n",
            "|    explained_variance | 0.439     |\n",
            "|    learning_rate      | 0.0007    |\n",
            "|    n_updates          | 1299      |\n",
            "|    policy_loss        | -32.1     |\n",
            "|    reward             | 2.2411277 |\n",
            "|    std                | 0.994     |\n",
            "|    value_loss         | 0.802     |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 98        |\n",
            "|    iterations         | 1400      |\n",
            "|    time_elapsed       | 70        |\n",
            "|    total_timesteps    | 7000      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -41       |\n",
            "|    explained_variance | 0.136     |\n",
            "|    learning_rate      | 0.0007    |\n",
            "|    n_updates          | 1399      |\n",
            "|    policy_loss        | 105       |\n",
            "|    reward             | -1.403822 |\n",
            "|    std                | 0.994     |\n",
            "|    value_loss         | 10        |\n",
            "-------------------------------------\n",
            "------------------------------------\n",
            "| time/                 |          |\n",
            "|    fps                | 98       |\n",
            "|    iterations         | 1500     |\n",
            "|    time_elapsed       | 76       |\n",
            "|    total_timesteps    | 7500     |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -40.9    |\n",
            "|    explained_variance | -0.1     |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 1499     |\n",
            "|    policy_loss        | 179      |\n",
            "|    reward             | 1.388676 |\n",
            "|    std                | 0.993    |\n",
            "|    value_loss         | 21.9     |\n",
            "------------------------------------\n",
            "--------------------------------------\n",
            "| time/                 |            |\n",
            "|    fps                | 98         |\n",
            "|    iterations         | 1600       |\n",
            "|    time_elapsed       | 81         |\n",
            "|    total_timesteps    | 8000       |\n",
            "| train/                |            |\n",
            "|    entropy_loss       | -40.9      |\n",
            "|    explained_variance | 0          |\n",
            "|    learning_rate      | 0.0007     |\n",
            "|    n_updates          | 1599       |\n",
            "|    policy_loss        | 122        |\n",
            "|    reward             | -1.1750767 |\n",
            "|    std                | 0.993      |\n",
            "|    value_loss         | 11.6       |\n",
            "--------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 98        |\n",
            "|    iterations         | 1700      |\n",
            "|    time_elapsed       | 86        |\n",
            "|    total_timesteps    | 8500      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -40.9     |\n",
            "|    explained_variance | 1.19e-07  |\n",
            "|    learning_rate      | 0.0007    |\n",
            "|    n_updates          | 1699      |\n",
            "|    policy_loss        | -644      |\n",
            "|    reward             | 5.2921677 |\n",
            "|    std                | 0.993     |\n",
            "|    value_loss         | 260       |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 98        |\n",
            "|    iterations         | 1800      |\n",
            "|    time_elapsed       | 91        |\n",
            "|    total_timesteps    | 9000      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -40.9     |\n",
            "|    explained_variance | -1.19e-07 |\n",
            "|    learning_rate      | 0.0007    |\n",
            "|    n_updates          | 1799      |\n",
            "|    policy_loss        | 674       |\n",
            "|    reward             | 3.8561575 |\n",
            "|    std                | 0.993     |\n",
            "|    value_loss         | 382       |\n",
            "-------------------------------------\n",
            "------------------------------------\n",
            "| time/                 |          |\n",
            "|    fps                | 98       |\n",
            "|    iterations         | 1900     |\n",
            "|    time_elapsed       | 96       |\n",
            "|    total_timesteps    | 9500     |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -41      |\n",
            "|    explained_variance | 0.107    |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 1899     |\n",
            "|    policy_loss        | 4.04     |\n",
            "|    reward             | 1.178899 |\n",
            "|    std                | 0.995    |\n",
            "|    value_loss         | 1.83     |\n",
            "------------------------------------\n",
            "--------------------------------------\n",
            "| time/                 |            |\n",
            "|    fps                | 98         |\n",
            "|    iterations         | 2000       |\n",
            "|    time_elapsed       | 101        |\n",
            "|    total_timesteps    | 10000      |\n",
            "| train/                |            |\n",
            "|    entropy_loss       | -41        |\n",
            "|    explained_variance | 0          |\n",
            "|    learning_rate      | 0.0007     |\n",
            "|    n_updates          | 1999       |\n",
            "|    policy_loss        | 91         |\n",
            "|    reward             | -0.9507761 |\n",
            "|    std                | 0.995      |\n",
            "|    value_loss         | 5.67       |\n",
            "--------------------------------------\n",
            "======a2c Validation from:  2022-01-03 to  2022-04-04\n",
            "a2c Sharpe Ratio:  -0.14881682635553525\n",
            "======ddpg Training========\n",
            "{'buffer_size': 10000, 'learning_rate': 0.0005, 'batch_size': 64}\n",
            "Using cuda device\n",
            "Logging to tensorboard_log/ddpg/ddpg_189_1\n",
            "day: 3020, episode: 5\n",
            "begin_total_asset: 1000000.00\n",
            "end_total_asset: 4134941.46\n",
            "total_reward: 3134941.46\n",
            "total_cost: 6531.55\n",
            "total_trades: 35218\n",
            "Sharpe: 0.730\n",
            "=================================\n",
            "======ddpg Validation from:  2022-01-03 to  2022-04-04\n",
            "ddpg Sharpe Ratio:  -0.23323576348385744\n",
            "======td3 Training========\n",
            "{'batch_size': 100, 'buffer_size': 1000000, 'learning_rate': 0.0001}\n",
            "Using cuda device\n",
            "Logging to tensorboard_log/td3/td3_189_1\n",
            "day: 3020, episode: 10\n",
            "begin_total_asset: 1000000.00\n",
            "end_total_asset: 6393593.18\n",
            "total_reward: 5393593.18\n",
            "total_cost: 1548.11\n",
            "total_trades: 66333\n",
            "Sharpe: 1.008\n",
            "=================================\n",
            "======td3 Validation from:  2022-01-03 to  2022-04-04\n",
            "td3 Sharpe Ratio:  -0.22726474272699887\n",
            "======sac Training========\n",
            "{'batch_size': 64, 'buffer_size': 100000, 'learning_rate': 0.0001, 'learning_starts': 100, 'ent_coef': 'auto_0.1'}\n",
            "Using cuda device\n",
            "Logging to tensorboard_log/sac/sac_189_1\n",
            "day: 3020, episode: 15\n",
            "begin_total_asset: 1000000.00\n",
            "end_total_asset: 4134489.29\n",
            "total_reward: 3134489.29\n",
            "total_cost: 268303.08\n",
            "total_trades: 69219\n",
            "Sharpe: 0.757\n",
            "=================================\n",
            "======sac Validation from:  2022-01-03 to  2022-04-04\n",
            "sac Sharpe Ratio:  -0.12984590976077562\n",
            "======ppo Training========\n",
            "{'ent_coef': 0.01, 'n_steps': 2048, 'learning_rate': 0.00025, 'batch_size': 128}\n",
            "Using cuda device\n",
            "Logging to tensorboard_log/ppo/ppo_189_1\n",
            "----------------------------------\n",
            "| time/              |           |\n",
            "|    fps             | 102       |\n",
            "|    iterations      | 1         |\n",
            "|    time_elapsed    | 19        |\n",
            "|    total_timesteps | 2048      |\n",
            "| train/             |           |\n",
            "|    reward          | 0.6000615 |\n",
            "----------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 100         |\n",
            "|    iterations           | 2           |\n",
            "|    time_elapsed         | 40          |\n",
            "|    total_timesteps      | 4096        |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.015176104 |\n",
            "|    clip_fraction        | 0.199       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -41.2       |\n",
            "|    explained_variance   | 0.0105      |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 4.55        |\n",
            "|    n_updates            | 10          |\n",
            "|    policy_gradient_loss | -0.0282     |\n",
            "|    reward               | -0.84197414 |\n",
            "|    std                  | 1           |\n",
            "|    value_loss           | 9.05        |\n",
            "-----------------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 101         |\n",
            "|    iterations           | 3           |\n",
            "|    time_elapsed         | 60          |\n",
            "|    total_timesteps      | 6144        |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.010344334 |\n",
            "|    clip_fraction        | 0.12        |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -41.3       |\n",
            "|    explained_variance   | -0.00571    |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 15.5        |\n",
            "|    n_updates            | 20          |\n",
            "|    policy_gradient_loss | -0.0171     |\n",
            "|    reward               | -0.52176756 |\n",
            "|    std                  | 1           |\n",
            "|    value_loss           | 44.1        |\n",
            "-----------------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 101         |\n",
            "|    iterations           | 4           |\n",
            "|    time_elapsed         | 80          |\n",
            "|    total_timesteps      | 8192        |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.013163242 |\n",
            "|    clip_fraction        | 0.146       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -41.3       |\n",
            "|    explained_variance   | -0.00859    |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 15.3        |\n",
            "|    n_updates            | 30          |\n",
            "|    policy_gradient_loss | -0.0224     |\n",
            "|    reward               | -2.1821563  |\n",
            "|    std                  | 1           |\n",
            "|    value_loss           | 45.5        |\n",
            "-----------------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 100         |\n",
            "|    iterations           | 5           |\n",
            "|    time_elapsed         | 101         |\n",
            "|    total_timesteps      | 10240       |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.014408065 |\n",
            "|    clip_fraction        | 0.187       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -41.3       |\n",
            "|    explained_variance   | -0.0553     |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 9.32        |\n",
            "|    n_updates            | 40          |\n",
            "|    policy_gradient_loss | -0.0217     |\n",
            "|    reward               | 0.40127692  |\n",
            "|    std                  | 1.01        |\n",
            "|    value_loss           | 16.7        |\n",
            "-----------------------------------------\n",
            "======ppo Validation from:  2022-01-03 to  2022-04-04\n",
            "ppo Sharpe Ratio:  -0.10733817017427076\n",
            "======Best Model Retraining from:  2010-01-01 to  2022-04-04\n",
            "======Trading from:  2022-04-04 to  2022-07-06\n",
            "============================================\n",
            "turbulence_threshold:  201.74162030011615\n",
            "======Model training from:  2010-01-01 to  2022-04-04\n",
            "======a2c Training========\n",
            "{'n_steps': 5, 'ent_coef': 0.005, 'learning_rate': 0.0007}\n",
            "Using cuda device\n",
            "Logging to tensorboard_log/a2c/a2c_252_1\n",
            "--------------------------------------\n",
            "| time/                 |            |\n",
            "|    fps                | 96         |\n",
            "|    iterations         | 100        |\n",
            "|    time_elapsed       | 5          |\n",
            "|    total_timesteps    | 500        |\n",
            "| train/                |            |\n",
            "|    entropy_loss       | -41.2      |\n",
            "|    explained_variance | 0          |\n",
            "|    learning_rate      | 0.0007     |\n",
            "|    n_updates          | 99         |\n",
            "|    policy_loss        | -38.8      |\n",
            "|    reward             | 0.15574819 |\n",
            "|    std                | 1          |\n",
            "|    value_loss         | 1.31       |\n",
            "--------------------------------------\n",
            "--------------------------------------\n",
            "| time/                 |            |\n",
            "|    fps                | 91         |\n",
            "|    iterations         | 200        |\n",
            "|    time_elapsed       | 10         |\n",
            "|    total_timesteps    | 1000       |\n",
            "| train/                |            |\n",
            "|    entropy_loss       | -41.1      |\n",
            "|    explained_variance | -0.0432    |\n",
            "|    learning_rate      | 0.0007     |\n",
            "|    n_updates          | 199        |\n",
            "|    policy_loss        | -27.8      |\n",
            "|    reward             | 0.48102596 |\n",
            "|    std                | 0.999      |\n",
            "|    value_loss         | 2.17       |\n",
            "--------------------------------------\n",
            "--------------------------------------\n",
            "| time/                 |            |\n",
            "|    fps                | 94         |\n",
            "|    iterations         | 300        |\n",
            "|    time_elapsed       | 15         |\n",
            "|    total_timesteps    | 1500       |\n",
            "| train/                |            |\n",
            "|    entropy_loss       | -41.1      |\n",
            "|    explained_variance | -0.0275    |\n",
            "|    learning_rate      | 0.0007     |\n",
            "|    n_updates          | 299        |\n",
            "|    policy_loss        | -32.8      |\n",
            "|    reward             | -2.3948188 |\n",
            "|    std                | 1          |\n",
            "|    value_loss         | 4.81       |\n",
            "--------------------------------------\n",
            "--------------------------------------\n",
            "| time/                 |            |\n",
            "|    fps                | 92         |\n",
            "|    iterations         | 400        |\n",
            "|    time_elapsed       | 21         |\n",
            "|    total_timesteps    | 2000       |\n",
            "| train/                |            |\n",
            "|    entropy_loss       | -41.1      |\n",
            "|    explained_variance | 0.113      |\n",
            "|    learning_rate      | 0.0007     |\n",
            "|    n_updates          | 399        |\n",
            "|    policy_loss        | -105       |\n",
            "|    reward             | -0.2406286 |\n",
            "|    std                | 1          |\n",
            "|    value_loss         | 12.2       |\n",
            "--------------------------------------\n",
            "--------------------------------------\n",
            "| time/                 |            |\n",
            "|    fps                | 94         |\n",
            "|    iterations         | 500        |\n",
            "|    time_elapsed       | 26         |\n",
            "|    total_timesteps    | 2500       |\n",
            "| train/                |            |\n",
            "|    entropy_loss       | -41.2      |\n",
            "|    explained_variance | 0          |\n",
            "|    learning_rate      | 0.0007     |\n",
            "|    n_updates          | 499        |\n",
            "|    policy_loss        | -83.1      |\n",
            "|    reward             | -1.0310625 |\n",
            "|    std                | 1          |\n",
            "|    value_loss         | 9.53       |\n",
            "--------------------------------------\n",
            "------------------------------------\n",
            "| time/                 |          |\n",
            "|    fps                | 93       |\n",
            "|    iterations         | 600      |\n",
            "|    time_elapsed       | 32       |\n",
            "|    total_timesteps    | 3000     |\n",
            "| train/                |          |\n",
            "|    entropy_loss       | -41.1    |\n",
            "|    explained_variance | 1.19e-07 |\n",
            "|    learning_rate      | 0.0007   |\n",
            "|    n_updates          | 599      |\n",
            "|    policy_loss        | -2.08    |\n",
            "|    reward             | 9.404537 |\n",
            "|    std                | 1        |\n",
            "|    value_loss         | 15.6     |\n",
            "------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 92        |\n",
            "|    iterations         | 700       |\n",
            "|    time_elapsed       | 37        |\n",
            "|    total_timesteps    | 3500      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -41.1     |\n",
            "|    explained_variance | -0.0019   |\n",
            "|    learning_rate      | 0.0007    |\n",
            "|    n_updates          | 699       |\n",
            "|    policy_loss        | -203      |\n",
            "|    reward             | 1.8093679 |\n",
            "|    std                | 1         |\n",
            "|    value_loss         | 27        |\n",
            "-------------------------------------\n",
            "--------------------------------------\n",
            "| time/                 |            |\n",
            "|    fps                | 93         |\n",
            "|    iterations         | 800        |\n",
            "|    time_elapsed       | 42         |\n",
            "|    total_timesteps    | 4000       |\n",
            "| train/                |            |\n",
            "|    entropy_loss       | -41.1      |\n",
            "|    explained_variance | 0          |\n",
            "|    learning_rate      | 0.0007     |\n",
            "|    n_updates          | 799        |\n",
            "|    policy_loss        | -155       |\n",
            "|    reward             | 0.29696387 |\n",
            "|    std                | 1          |\n",
            "|    value_loss         | 18.6       |\n",
            "--------------------------------------\n",
            "--------------------------------------\n",
            "| time/                 |            |\n",
            "|    fps                | 92         |\n",
            "|    iterations         | 900        |\n",
            "|    time_elapsed       | 48         |\n",
            "|    total_timesteps    | 4500       |\n",
            "| train/                |            |\n",
            "|    entropy_loss       | -41.1      |\n",
            "|    explained_variance | 5.96e-08   |\n",
            "|    learning_rate      | 0.0007     |\n",
            "|    n_updates          | 899        |\n",
            "|    policy_loss        | -86.9      |\n",
            "|    reward             | -1.6688259 |\n",
            "|    std                | 1          |\n",
            "|    value_loss         | 8.07       |\n",
            "--------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 92        |\n",
            "|    iterations         | 1000      |\n",
            "|    time_elapsed       | 54        |\n",
            "|    total_timesteps    | 5000      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -41.1     |\n",
            "|    explained_variance | -2.38e-07 |\n",
            "|    learning_rate      | 0.0007    |\n",
            "|    n_updates          | 999       |\n",
            "|    policy_loss        | 109       |\n",
            "|    reward             | 3.7493455 |\n",
            "|    std                | 1         |\n",
            "|    value_loss         | 15.6      |\n",
            "-------------------------------------\n",
            "--------------------------------------\n",
            "| time/                 |            |\n",
            "|    fps                | 91         |\n",
            "|    iterations         | 1100       |\n",
            "|    time_elapsed       | 60         |\n",
            "|    total_timesteps    | 5500       |\n",
            "| train/                |            |\n",
            "|    entropy_loss       | -41.1      |\n",
            "|    explained_variance | 0          |\n",
            "|    learning_rate      | 0.0007     |\n",
            "|    n_updates          | 1099       |\n",
            "|    policy_loss        | -399       |\n",
            "|    reward             | -1.8209955 |\n",
            "|    std                | 0.999      |\n",
            "|    value_loss         | 126        |\n",
            "--------------------------------------\n",
            "--------------------------------------\n",
            "| time/                 |            |\n",
            "|    fps                | 92         |\n",
            "|    iterations         | 1200       |\n",
            "|    time_elapsed       | 65         |\n",
            "|    total_timesteps    | 6000       |\n",
            "| train/                |            |\n",
            "|    entropy_loss       | -41.1      |\n",
            "|    explained_variance | 0          |\n",
            "|    learning_rate      | 0.0007     |\n",
            "|    n_updates          | 1199       |\n",
            "|    policy_loss        | -121       |\n",
            "|    reward             | -5.2913017 |\n",
            "|    std                | 0.998      |\n",
            "|    value_loss         | 24         |\n",
            "--------------------------------------\n",
            "---------------------------------------\n",
            "| time/                 |             |\n",
            "|    fps                | 91          |\n",
            "|    iterations         | 1300        |\n",
            "|    time_elapsed       | 70          |\n",
            "|    total_timesteps    | 6500        |\n",
            "| train/                |             |\n",
            "|    entropy_loss       | -41.1       |\n",
            "|    explained_variance | 0.139       |\n",
            "|    learning_rate      | 0.0007      |\n",
            "|    n_updates          | 1299        |\n",
            "|    policy_loss        | 164         |\n",
            "|    reward             | -0.32371435 |\n",
            "|    std                | 1           |\n",
            "|    value_loss         | 16.9        |\n",
            "---------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 92        |\n",
            "|    iterations         | 1400      |\n",
            "|    time_elapsed       | 75        |\n",
            "|    total_timesteps    | 7000      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -41.1     |\n",
            "|    explained_variance | -0.0498   |\n",
            "|    learning_rate      | 0.0007    |\n",
            "|    n_updates          | 1399      |\n",
            "|    policy_loss        | -12       |\n",
            "|    reward             | 0.8689141 |\n",
            "|    std                | 1         |\n",
            "|    value_loss         | 2.3       |\n",
            "-------------------------------------\n",
            "---------------------------------------\n",
            "| time/                 |             |\n",
            "|    fps                | 92          |\n",
            "|    iterations         | 1500        |\n",
            "|    time_elapsed       | 80          |\n",
            "|    total_timesteps    | 7500        |\n",
            "| train/                |             |\n",
            "|    entropy_loss       | -41.1       |\n",
            "|    explained_variance | 0           |\n",
            "|    learning_rate      | 0.0007      |\n",
            "|    n_updates          | 1499        |\n",
            "|    policy_loss        | -72.2       |\n",
            "|    reward             | -0.54522103 |\n",
            "|    std                | 1           |\n",
            "|    value_loss         | 3.61        |\n",
            "---------------------------------------\n",
            "--------------------------------------\n",
            "| time/                 |            |\n",
            "|    fps                | 92         |\n",
            "|    iterations         | 1600       |\n",
            "|    time_elapsed       | 86         |\n",
            "|    total_timesteps    | 8000       |\n",
            "| train/                |            |\n",
            "|    entropy_loss       | -41.2      |\n",
            "|    explained_variance | -2.38e-07  |\n",
            "|    learning_rate      | 0.0007     |\n",
            "|    n_updates          | 1599       |\n",
            "|    policy_loss        | -13.1      |\n",
            "|    reward             | -1.0812243 |\n",
            "|    std                | 1          |\n",
            "|    value_loss         | 0.389      |\n",
            "--------------------------------------\n",
            "---------------------------------------\n",
            "| time/                 |             |\n",
            "|    fps                | 93          |\n",
            "|    iterations         | 1700        |\n",
            "|    time_elapsed       | 91          |\n",
            "|    total_timesteps    | 8500        |\n",
            "| train/                |             |\n",
            "|    entropy_loss       | -41.2       |\n",
            "|    explained_variance | -1.19e-07   |\n",
            "|    learning_rate      | 0.0007      |\n",
            "|    n_updates          | 1699        |\n",
            "|    policy_loss        | -86.9       |\n",
            "|    reward             | -0.64294523 |\n",
            "|    std                | 1           |\n",
            "|    value_loss         | 5.99        |\n",
            "---------------------------------------\n",
            "--------------------------------------\n",
            "| time/                 |            |\n",
            "|    fps                | 92         |\n",
            "|    iterations         | 1800       |\n",
            "|    time_elapsed       | 97         |\n",
            "|    total_timesteps    | 9000       |\n",
            "| train/                |            |\n",
            "|    entropy_loss       | -41.2      |\n",
            "|    explained_variance | -1.19e-07  |\n",
            "|    learning_rate      | 0.0007     |\n",
            "|    n_updates          | 1799       |\n",
            "|    policy_loss        | 413        |\n",
            "|    reward             | 0.50728357 |\n",
            "|    std                | 1          |\n",
            "|    value_loss         | 127        |\n",
            "--------------------------------------\n",
            "---------------------------------------\n",
            "| time/                 |             |\n",
            "|    fps                | 93          |\n",
            "|    iterations         | 1900        |\n",
            "|    time_elapsed       | 102         |\n",
            "|    total_timesteps    | 9500        |\n",
            "| train/                |             |\n",
            "|    entropy_loss       | -41.2       |\n",
            "|    explained_variance | 0           |\n",
            "|    learning_rate      | 0.0007      |\n",
            "|    n_updates          | 1899        |\n",
            "|    policy_loss        | -55.7       |\n",
            "|    reward             | -0.07039916 |\n",
            "|    std                | 1           |\n",
            "|    value_loss         | 2.31        |\n",
            "---------------------------------------\n",
            "--------------------------------------\n",
            "| time/                 |            |\n",
            "|    fps                | 92         |\n",
            "|    iterations         | 2000       |\n",
            "|    time_elapsed       | 107        |\n",
            "|    total_timesteps    | 10000      |\n",
            "| train/                |            |\n",
            "|    entropy_loss       | -41.2      |\n",
            "|    explained_variance | 0          |\n",
            "|    learning_rate      | 0.0007     |\n",
            "|    n_updates          | 1999       |\n",
            "|    policy_loss        | -22.2      |\n",
            "|    reward             | -0.5126278 |\n",
            "|    std                | 1          |\n",
            "|    value_loss         | 0.961      |\n",
            "--------------------------------------\n",
            "======a2c Validation from:  2022-04-04 to  2022-07-06\n",
            "a2c Sharpe Ratio:  -0.25366636627181594\n",
            "======ddpg Training========\n",
            "{'buffer_size': 10000, 'learning_rate': 0.0005, 'batch_size': 64}\n",
            "Using cuda device\n",
            "Logging to tensorboard_log/ddpg/ddpg_252_1\n",
            "day: 3083, episode: 5\n",
            "begin_total_asset: 1000000.00\n",
            "end_total_asset: 5721518.02\n",
            "total_reward: 4721518.02\n",
            "total_cost: 6164.84\n",
            "total_trades: 44663\n",
            "Sharpe: 0.868\n",
            "=================================\n",
            "======ddpg Validation from:  2022-04-04 to  2022-07-06\n",
            "ddpg Sharpe Ratio:  -0.22747056221011977\n",
            "======td3 Training========\n",
            "{'batch_size': 100, 'buffer_size': 1000000, 'learning_rate': 0.0001}\n",
            "Using cuda device\n",
            "Logging to tensorboard_log/td3/td3_252_1\n",
            "day: 3083, episode: 10\n",
            "begin_total_asset: 1000000.00\n",
            "end_total_asset: 4419882.67\n",
            "total_reward: 3419882.67\n",
            "total_cost: 1787.44\n",
            "total_trades: 46447\n",
            "Sharpe: 0.822\n",
            "=================================\n",
            "======td3 Validation from:  2022-04-04 to  2022-07-06\n",
            "td3 Sharpe Ratio:  -0.2839424978995499\n",
            "======sac Training========\n",
            "{'batch_size': 64, 'buffer_size': 100000, 'learning_rate': 0.0001, 'learning_starts': 100, 'ent_coef': 'auto_0.1'}\n",
            "Using cuda device\n",
            "Logging to tensorboard_log/sac/sac_252_1\n",
            "day: 3083, episode: 15\n",
            "begin_total_asset: 1000000.00\n",
            "end_total_asset: 4927709.27\n",
            "total_reward: 3927709.27\n",
            "total_cost: 85026.93\n",
            "total_trades: 55788\n",
            "Sharpe: 0.808\n",
            "=================================\n",
            "======sac Validation from:  2022-04-04 to  2022-07-06\n",
            "sac Sharpe Ratio:  -0.16022717745791382\n",
            "======ppo Training========\n",
            "{'ent_coef': 0.01, 'n_steps': 2048, 'learning_rate': 0.00025, 'batch_size': 128}\n",
            "Using cuda device\n",
            "Logging to tensorboard_log/ppo/ppo_252_1\n",
            "-----------------------------------\n",
            "| time/              |            |\n",
            "|    fps             | 105        |\n",
            "|    iterations      | 1          |\n",
            "|    time_elapsed    | 19         |\n",
            "|    total_timesteps | 2048       |\n",
            "| train/             |            |\n",
            "|    reward          | 0.54891366 |\n",
            "-----------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 103         |\n",
            "|    iterations           | 2           |\n",
            "|    time_elapsed         | 39          |\n",
            "|    total_timesteps      | 4096        |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.016892547 |\n",
            "|    clip_fraction        | 0.209       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -41.2       |\n",
            "|    explained_variance   | -0.0182     |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 5.17        |\n",
            "|    n_updates            | 10          |\n",
            "|    policy_gradient_loss | -0.0246     |\n",
            "|    reward               | -0.46093306 |\n",
            "|    std                  | 1           |\n",
            "|    value_loss           | 10.4        |\n",
            "-----------------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 104         |\n",
            "|    iterations           | 3           |\n",
            "|    time_elapsed         | 58          |\n",
            "|    total_timesteps      | 6144        |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.013119971 |\n",
            "|    clip_fraction        | 0.109       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -41.2       |\n",
            "|    explained_variance   | -0.000288   |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 28.6        |\n",
            "|    n_updates            | 20          |\n",
            "|    policy_gradient_loss | -0.0202     |\n",
            "|    reward               | -9.113171   |\n",
            "|    std                  | 1           |\n",
            "|    value_loss           | 57.9        |\n",
            "-----------------------------------------\n",
            "------------------------------------------\n",
            "| time/                   |              |\n",
            "|    fps                  | 102          |\n",
            "|    iterations           | 4            |\n",
            "|    time_elapsed         | 79           |\n",
            "|    total_timesteps      | 8192         |\n",
            "| train/                  |              |\n",
            "|    approx_kl            | 0.0148056755 |\n",
            "|    clip_fraction        | 0.147        |\n",
            "|    clip_range           | 0.2          |\n",
            "|    entropy_loss         | -41.3        |\n",
            "|    explained_variance   | -0.0199      |\n",
            "|    learning_rate        | 0.00025      |\n",
            "|    loss                 | 47.2         |\n",
            "|    n_updates            | 30           |\n",
            "|    policy_gradient_loss | -0.0206      |\n",
            "|    reward               | -0.5656307   |\n",
            "|    std                  | 1.01         |\n",
            "|    value_loss           | 78.6         |\n",
            "------------------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 102         |\n",
            "|    iterations           | 5           |\n",
            "|    time_elapsed         | 99          |\n",
            "|    total_timesteps      | 10240       |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.027241705 |\n",
            "|    clip_fraction        | 0.268       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -41.3       |\n",
            "|    explained_variance   | -0.00487    |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 8.49        |\n",
            "|    n_updates            | 40          |\n",
            "|    policy_gradient_loss | -0.0307     |\n",
            "|    reward               | -0.5270617  |\n",
            "|    std                  | 1.01        |\n",
            "|    value_loss           | 17.2        |\n",
            "-----------------------------------------\n",
            "======ppo Validation from:  2022-04-04 to  2022-07-06\n",
            "ppo Sharpe Ratio:  -0.23906732813732043\n",
            "======Best Model Retraining from:  2010-01-01 to  2022-07-06\n",
            "======Trading from:  2022-07-06 to  2022-10-04\n",
            "============================================\n",
            "turbulence_threshold:  201.74162030011615\n",
            "======Model training from:  2010-01-01 to  2022-07-06\n",
            "======a2c Training========\n",
            "{'n_steps': 5, 'ent_coef': 0.005, 'learning_rate': 0.0007}\n",
            "Using cuda device\n",
            "Logging to tensorboard_log/a2c/a2c_315_1\n",
            "--------------------------------------\n",
            "| time/                 |            |\n",
            "|    fps                | 102        |\n",
            "|    iterations         | 100        |\n",
            "|    time_elapsed       | 4          |\n",
            "|    total_timesteps    | 500        |\n",
            "| train/                |            |\n",
            "|    entropy_loss       | -41.2      |\n",
            "|    explained_variance | 0.0353     |\n",
            "|    learning_rate      | 0.0007     |\n",
            "|    n_updates          | 99         |\n",
            "|    policy_loss        | -80        |\n",
            "|    reward             | 0.24549441 |\n",
            "|    std                | 1          |\n",
            "|    value_loss         | 4.92       |\n",
            "--------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 97        |\n",
            "|    iterations         | 200       |\n",
            "|    time_elapsed       | 10        |\n",
            "|    total_timesteps    | 1000      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -41.1     |\n",
            "|    explained_variance | -0.126    |\n",
            "|    learning_rate      | 0.0007    |\n",
            "|    n_updates          | 199       |\n",
            "|    policy_loss        | -17.6     |\n",
            "|    reward             | 1.3894979 |\n",
            "|    std                | 1         |\n",
            "|    value_loss         | 3.75      |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 96        |\n",
            "|    iterations         | 300       |\n",
            "|    time_elapsed       | 15        |\n",
            "|    total_timesteps    | 1500      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -41.2     |\n",
            "|    explained_variance | -0.012    |\n",
            "|    learning_rate      | 0.0007    |\n",
            "|    n_updates          | 299       |\n",
            "|    policy_loss        | -91.5     |\n",
            "|    reward             | -2.606353 |\n",
            "|    std                | 1         |\n",
            "|    value_loss         | 8.84      |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 97        |\n",
            "|    iterations         | 400       |\n",
            "|    time_elapsed       | 20        |\n",
            "|    total_timesteps    | 2000      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -41.2     |\n",
            "|    explained_variance | -0.183    |\n",
            "|    learning_rate      | 0.0007    |\n",
            "|    n_updates          | 399       |\n",
            "|    policy_loss        | -10.2     |\n",
            "|    reward             | 1.6483078 |\n",
            "|    std                | 1         |\n",
            "|    value_loss         | 4.13      |\n",
            "-------------------------------------\n",
            "---------------------------------------\n",
            "| time/                 |             |\n",
            "|    fps                | 95          |\n",
            "|    iterations         | 500         |\n",
            "|    time_elapsed       | 26          |\n",
            "|    total_timesteps    | 2500        |\n",
            "| train/                |             |\n",
            "|    entropy_loss       | -41.2       |\n",
            "|    explained_variance | 0.0521      |\n",
            "|    learning_rate      | 0.0007      |\n",
            "|    n_updates          | 499         |\n",
            "|    policy_loss        | -51.2       |\n",
            "|    reward             | -0.46745828 |\n",
            "|    std                | 1           |\n",
            "|    value_loss         | 2.93        |\n",
            "---------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 96        |\n",
            "|    iterations         | 600       |\n",
            "|    time_elapsed       | 31        |\n",
            "|    total_timesteps    | 3000      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -41.2     |\n",
            "|    explained_variance | 0.0325    |\n",
            "|    learning_rate      | 0.0007    |\n",
            "|    n_updates          | 599       |\n",
            "|    policy_loss        | -2.87     |\n",
            "|    reward             | 10.919484 |\n",
            "|    std                | 1         |\n",
            "|    value_loss         | 8.18      |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 94        |\n",
            "|    iterations         | 700       |\n",
            "|    time_elapsed       | 36        |\n",
            "|    total_timesteps    | 3500      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -41.3     |\n",
            "|    explained_variance | 0.0171    |\n",
            "|    learning_rate      | 0.0007    |\n",
            "|    n_updates          | 699       |\n",
            "|    policy_loss        | -27.7     |\n",
            "|    reward             | 0.8419629 |\n",
            "|    std                | 1.01      |\n",
            "|    value_loss         | 0.606     |\n",
            "-------------------------------------\n",
            "---------------------------------------\n",
            "| time/                 |             |\n",
            "|    fps                | 95          |\n",
            "|    iterations         | 800         |\n",
            "|    time_elapsed       | 41          |\n",
            "|    total_timesteps    | 4000        |\n",
            "| train/                |             |\n",
            "|    entropy_loss       | -41.4       |\n",
            "|    explained_variance | -1.19e-07   |\n",
            "|    learning_rate      | 0.0007      |\n",
            "|    n_updates          | 799         |\n",
            "|    policy_loss        | -27.5       |\n",
            "|    reward             | -0.18779811 |\n",
            "|    std                | 1.01        |\n",
            "|    value_loss         | 0.786       |\n",
            "---------------------------------------\n",
            "--------------------------------------\n",
            "| time/                 |            |\n",
            "|    fps                | 93         |\n",
            "|    iterations         | 900        |\n",
            "|    time_elapsed       | 47         |\n",
            "|    total_timesteps    | 4500       |\n",
            "| train/                |            |\n",
            "|    entropy_loss       | -41.3      |\n",
            "|    explained_variance | 1.19e-07   |\n",
            "|    learning_rate      | 0.0007     |\n",
            "|    n_updates          | 899        |\n",
            "|    policy_loss        | -23.3      |\n",
            "|    reward             | 0.21865338 |\n",
            "|    std                | 1.01       |\n",
            "|    value_loss         | 1.68       |\n",
            "--------------------------------------\n",
            "--------------------------------------\n",
            "| time/                 |            |\n",
            "|    fps                | 94         |\n",
            "|    iterations         | 1000       |\n",
            "|    time_elapsed       | 52         |\n",
            "|    total_timesteps    | 5000       |\n",
            "| train/                |            |\n",
            "|    entropy_loss       | -41.3      |\n",
            "|    explained_variance | 1.19e-07   |\n",
            "|    learning_rate      | 0.0007     |\n",
            "|    n_updates          | 999        |\n",
            "|    policy_loss        | 30.7       |\n",
            "|    reward             | -0.8657269 |\n",
            "|    std                | 1.01       |\n",
            "|    value_loss         | 0.856      |\n",
            "--------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 95        |\n",
            "|    iterations         | 1100      |\n",
            "|    time_elapsed       | 57        |\n",
            "|    total_timesteps    | 5500      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -41.4     |\n",
            "|    explained_variance | 5.96e-08  |\n",
            "|    learning_rate      | 0.0007    |\n",
            "|    n_updates          | 1099      |\n",
            "|    policy_loss        | 75.6      |\n",
            "|    reward             | 3.6971135 |\n",
            "|    std                | 1.01      |\n",
            "|    value_loss         | 7.44      |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 94        |\n",
            "|    iterations         | 1200      |\n",
            "|    time_elapsed       | 63        |\n",
            "|    total_timesteps    | 6000      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -41.3     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0007    |\n",
            "|    n_updates          | 1199      |\n",
            "|    policy_loss        | -237      |\n",
            "|    reward             | 0.9042094 |\n",
            "|    std                | 1.01      |\n",
            "|    value_loss         | 34.5      |\n",
            "-------------------------------------\n",
            "--------------------------------------\n",
            "| time/                 |            |\n",
            "|    fps                | 95         |\n",
            "|    iterations         | 1300       |\n",
            "|    time_elapsed       | 68         |\n",
            "|    total_timesteps    | 6500       |\n",
            "| train/                |            |\n",
            "|    entropy_loss       | -41.4      |\n",
            "|    explained_variance | -0.503     |\n",
            "|    learning_rate      | 0.0007     |\n",
            "|    n_updates          | 1299       |\n",
            "|    policy_loss        | 58.3       |\n",
            "|    reward             | 0.20865634 |\n",
            "|    std                | 1.01       |\n",
            "|    value_loss         | 2.1        |\n",
            "--------------------------------------\n",
            "--------------------------------------\n",
            "| time/                 |            |\n",
            "|    fps                | 94         |\n",
            "|    iterations         | 1400       |\n",
            "|    time_elapsed       | 74         |\n",
            "|    total_timesteps    | 7000       |\n",
            "| train/                |            |\n",
            "|    entropy_loss       | -41.3      |\n",
            "|    explained_variance | -1.19e-07  |\n",
            "|    learning_rate      | 0.0007     |\n",
            "|    n_updates          | 1399       |\n",
            "|    policy_loss        | -72.9      |\n",
            "|    reward             | -1.3368342 |\n",
            "|    std                | 1.01       |\n",
            "|    value_loss         | 4.41       |\n",
            "--------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 95        |\n",
            "|    iterations         | 1500      |\n",
            "|    time_elapsed       | 78        |\n",
            "|    total_timesteps    | 7500      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -41.4     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0007    |\n",
            "|    n_updates          | 1499      |\n",
            "|    policy_loss        | -144      |\n",
            "|    reward             | 3.5003781 |\n",
            "|    std                | 1.01      |\n",
            "|    value_loss         | 14.4      |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 94        |\n",
            "|    iterations         | 1600      |\n",
            "|    time_elapsed       | 84        |\n",
            "|    total_timesteps    | 8000      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -41.4     |\n",
            "|    explained_variance | 1.19e-07  |\n",
            "|    learning_rate      | 0.0007    |\n",
            "|    n_updates          | 1599      |\n",
            "|    policy_loss        | -38.2     |\n",
            "|    reward             | 1.3683945 |\n",
            "|    std                | 1.01      |\n",
            "|    value_loss         | 1.89      |\n",
            "-------------------------------------\n",
            "-------------------------------------\n",
            "| time/                 |           |\n",
            "|    fps                | 95        |\n",
            "|    iterations         | 1700      |\n",
            "|    time_elapsed       | 89        |\n",
            "|    total_timesteps    | 8500      |\n",
            "| train/                |           |\n",
            "|    entropy_loss       | -41.4     |\n",
            "|    explained_variance | 0         |\n",
            "|    learning_rate      | 0.0007    |\n",
            "|    n_updates          | 1699      |\n",
            "|    policy_loss        | 48.4      |\n",
            "|    reward             | 1.4903697 |\n",
            "|    std                | 1.01      |\n",
            "|    value_loss         | 4.15      |\n",
            "-------------------------------------\n",
            "--------------------------------------\n",
            "| time/                 |            |\n",
            "|    fps                | 95         |\n",
            "|    iterations         | 1800       |\n",
            "|    time_elapsed       | 94         |\n",
            "|    total_timesteps    | 9000       |\n",
            "| train/                |            |\n",
            "|    entropy_loss       | -41.4      |\n",
            "|    explained_variance | -1.19e-07  |\n",
            "|    learning_rate      | 0.0007     |\n",
            "|    n_updates          | 1799       |\n",
            "|    policy_loss        | -31.4      |\n",
            "|    reward             | -1.6481568 |\n",
            "|    std                | 1.01       |\n",
            "|    value_loss         | 17.5       |\n",
            "--------------------------------------\n",
            "---------------------------------------\n",
            "| time/                 |             |\n",
            "|    fps                | 95          |\n",
            "|    iterations         | 1900        |\n",
            "|    time_elapsed       | 99          |\n",
            "|    total_timesteps    | 9500        |\n",
            "| train/                |             |\n",
            "|    entropy_loss       | -41.4       |\n",
            "|    explained_variance | 0           |\n",
            "|    learning_rate      | 0.0007      |\n",
            "|    n_updates          | 1899        |\n",
            "|    policy_loss        | 27.9        |\n",
            "|    reward             | -0.39915258 |\n",
            "|    std                | 1.01        |\n",
            "|    value_loss         | 0.507       |\n",
            "---------------------------------------\n",
            "--------------------------------------\n",
            "| time/                 |            |\n",
            "|    fps                | 95         |\n",
            "|    iterations         | 2000       |\n",
            "|    time_elapsed       | 104        |\n",
            "|    total_timesteps    | 10000      |\n",
            "| train/                |            |\n",
            "|    entropy_loss       | -41.3      |\n",
            "|    explained_variance | 0.159      |\n",
            "|    learning_rate      | 0.0007     |\n",
            "|    n_updates          | 1999       |\n",
            "|    policy_loss        | 58.6       |\n",
            "|    reward             | -1.4694927 |\n",
            "|    std                | 1.01       |\n",
            "|    value_loss         | 2.85       |\n",
            "--------------------------------------\n",
            "======a2c Validation from:  2022-07-06 to  2022-10-04\n",
            "a2c Sharpe Ratio:  -0.10266785475978492\n",
            "======ddpg Training========\n",
            "{'buffer_size': 10000, 'learning_rate': 0.0005, 'batch_size': 64}\n",
            "Using cuda device\n",
            "Logging to tensorboard_log/ddpg/ddpg_315_1\n",
            "day: 3146, episode: 5\n",
            "begin_total_asset: 1000000.00\n",
            "end_total_asset: 8621242.15\n",
            "total_reward: 7621242.15\n",
            "total_cost: 7912.99\n",
            "total_trades: 39562\n",
            "Sharpe: 1.023\n",
            "=================================\n",
            "======ddpg Validation from:  2022-07-06 to  2022-10-04\n",
            "ddpg Sharpe Ratio:  -0.06187703782204383\n",
            "======td3 Training========\n",
            "{'batch_size': 100, 'buffer_size': 1000000, 'learning_rate': 0.0001}\n",
            "Using cuda device\n",
            "Logging to tensorboard_log/td3/td3_315_1\n",
            "day: 3146, episode: 10\n",
            "begin_total_asset: 1000000.00\n",
            "end_total_asset: 4437212.38\n",
            "total_reward: 3437212.38\n",
            "total_cost: 2996.04\n",
            "total_trades: 31813\n",
            "Sharpe: 0.776\n",
            "=================================\n",
            "======td3 Validation from:  2022-07-06 to  2022-10-04\n",
            "td3 Sharpe Ratio:  -0.12530693561038414\n",
            "======sac Training========\n",
            "{'batch_size': 64, 'buffer_size': 100000, 'learning_rate': 0.0001, 'learning_starts': 100, 'ent_coef': 'auto_0.1'}\n",
            "Using cuda device\n",
            "Logging to tensorboard_log/sac/sac_315_1\n",
            "day: 3146, episode: 15\n",
            "begin_total_asset: 1000000.00\n",
            "end_total_asset: 3601294.13\n",
            "total_reward: 2601294.13\n",
            "total_cost: 234860.64\n",
            "total_trades: 64468\n",
            "Sharpe: 0.695\n",
            "=================================\n",
            "======sac Validation from:  2022-07-06 to  2022-10-04\n",
            "sac Sharpe Ratio:  -0.16088947893289524\n",
            "======ppo Training========\n",
            "{'ent_coef': 0.01, 'n_steps': 2048, 'learning_rate': 0.00025, 'batch_size': 128}\n",
            "Using cuda device\n",
            "Logging to tensorboard_log/ppo/ppo_315_1\n",
            "----------------------------------\n",
            "| time/              |           |\n",
            "|    fps             | 105       |\n",
            "|    iterations      | 1         |\n",
            "|    time_elapsed    | 19        |\n",
            "|    total_timesteps | 2048      |\n",
            "| train/             |           |\n",
            "|    reward          | 1.6173023 |\n",
            "----------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 103         |\n",
            "|    iterations           | 2           |\n",
            "|    time_elapsed         | 39          |\n",
            "|    total_timesteps      | 4096        |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.015555255 |\n",
            "|    clip_fraction        | 0.228       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -41.2       |\n",
            "|    explained_variance   | -0.0101     |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 6.3         |\n",
            "|    n_updates            | 10          |\n",
            "|    policy_gradient_loss | -0.0255     |\n",
            "|    reward               | 3.5431957   |\n",
            "|    std                  | 1           |\n",
            "|    value_loss           | 13.2        |\n",
            "-----------------------------------------\n",
            "-----------------------------------------\n",
            "| time/                   |             |\n",
            "|    fps                  | 104         |\n",
            "|    iterations           | 3           |\n",
            "|    time_elapsed         | 58          |\n",
            "|    total_timesteps      | 6144        |\n",
            "| train/                  |             |\n",
            "|    approx_kl            | 0.015205141 |\n",
            "|    clip_fraction        | 0.167       |\n",
            "|    clip_range           | 0.2         |\n",
            "|    entropy_loss         | -41.2       |\n",
            "|    explained_variance   | 0.00121     |\n",
            "|    learning_rate        | 0.00025     |\n",
            "|    loss                 | 24.5        |\n",
            "|    n_updates            | 20          |\n",
            "|    policy_gradient_loss | -0.0202     |\n",
            "|    reward               | 5.9619627   |\n",
            "|    std                  | 1           |\n",
            "|    value_loss           | 81.8        |\n",
            "-----------------------------------------\n",
            "----------------------------------------\n",
            "| time/                   |            |\n",
            "|    fps                  | 103        |\n",
            "|    iterations           | 4          |\n",
            "|    time_elapsed         | 78         |\n",
            "|    total_timesteps      | 8192       |\n",
            "| train/                  |            |\n",
            "|    approx_kl            | 0.01795656 |\n",
            "|    clip_fraction        | 0.154      |\n",
            "|    clip_range           | 0.2        |\n",
            "|    entropy_loss         | -41.3      |\n",
            "|    explained_variance   | -0.00243   |\n",
            "|    learning_rate        | 0.00025    |\n",
            "|    loss                 | 31.6       |\n",
            "|    n_updates            | 30         |\n",
            "|    policy_gradient_loss | -0.0196    |\n",
            "|    reward               | 1.3367934  |\n",
            "|    std                  | 1.01       |\n",
            "|    value_loss           | 49.2       |\n",
            "----------------------------------------\n"
          ]
        }
      ],
      "source": [
        "df_summary = ensemble_agent.run_ensemble_strategy(A2C_model_kwargs,\n",
        "                                                 PPO_model_kwargs,\n",
        "                                                 DDPG_model_kwargs,\n",
        "                                                 SAC_model_kwargs,\n",
        "                                                 TD3_model_kwargs,\n",
        "                                                 timesteps_dict)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "id": "-0qd8acMtj1f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 89
        },
        "outputId": "ef19ff5d-9173-4268-e50b-6128cf9278f5"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Empty DataFrame\n",
              "Columns: [Iter, Val Start, Val End, Model Used, A2C Sharpe, PPO Sharpe, DDPG Sharpe, SAC Sharpe, TD3 Sharpe]\n",
              "Index: []"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-54aae13f-d8ce-4faf-8e3e-37668f9389e6\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Iter</th>\n",
              "      <th>Val Start</th>\n",
              "      <th>Val End</th>\n",
              "      <th>Model Used</th>\n",
              "      <th>A2C Sharpe</th>\n",
              "      <th>PPO Sharpe</th>\n",
              "      <th>DDPG Sharpe</th>\n",
              "      <th>SAC Sharpe</th>\n",
              "      <th>TD3 Sharpe</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-54aae13f-d8ce-4faf-8e3e-37668f9389e6')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-54aae13f-d8ce-4faf-8e3e-37668f9389e6 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-54aae13f-d8ce-4faf-8e3e-37668f9389e6');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "  <div id=\"id_6bb85869-9394-439e-9d2c-2fa4c43b0b65\">\n",
              "    <style>\n",
              "      .colab-df-generate {\n",
              "        background-color: #E8F0FE;\n",
              "        border: none;\n",
              "        border-radius: 50%;\n",
              "        cursor: pointer;\n",
              "        display: none;\n",
              "        fill: #1967D2;\n",
              "        height: 32px;\n",
              "        padding: 0 0 0 0;\n",
              "        width: 32px;\n",
              "      }\n",
              "\n",
              "      .colab-df-generate:hover {\n",
              "        background-color: #E2EBFA;\n",
              "        box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "        fill: #174EA6;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate {\n",
              "        background-color: #3B4455;\n",
              "        fill: #D2E3FC;\n",
              "      }\n",
              "\n",
              "      [theme=dark] .colab-df-generate:hover {\n",
              "        background-color: #434B5C;\n",
              "        box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "        filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "        fill: #FFFFFF;\n",
              "      }\n",
              "    </style>\n",
              "    <button class=\"colab-df-generate\" onclick=\"generateWithVariable('df_summary')\"\n",
              "            title=\"Generate code using this dataframe.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M7,19H8.4L18.45,9,17,7.55,7,17.6ZM5,21V16.75L18.45,3.32a2,2,0,0,1,2.83,0l1.4,1.43a1.91,1.91,0,0,1,.58,1.4,1.91,1.91,0,0,1-.58,1.4L9.25,21ZM18.45,9,17,7.55Zm-12,3A5.31,5.31,0,0,0,4.9,8.1,5.31,5.31,0,0,0,1,6.5,5.31,5.31,0,0,0,4.9,4.9,5.31,5.31,0,0,0,6.5,1,5.31,5.31,0,0,0,8.1,4.9,5.31,5.31,0,0,0,12,6.5,5.46,5.46,0,0,0,6.5,12Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "    <script>\n",
              "      (() => {\n",
              "      const buttonEl =\n",
              "        document.querySelector('#id_6bb85869-9394-439e-9d2c-2fa4c43b0b65 button.colab-df-generate');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      buttonEl.onclick = () => {\n",
              "        google.colab.notebook.generateWithVariable('df_summary');\n",
              "      }\n",
              "      })();\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "dataframe",
              "variable_name": "df_summary",
              "summary": "{\n  \"name\": \"df_summary\",\n  \"rows\": 0,\n  \"fields\": [\n    {\n      \"column\": \"Iter\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": null,\n        \"min\": null,\n        \"max\": null,\n        \"num_unique_values\": 0,\n        \"samples\": [],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Val Start\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": null,\n        \"min\": null,\n        \"max\": null,\n        \"num_unique_values\": 0,\n        \"samples\": [],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Val End\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": null,\n        \"min\": null,\n        \"max\": null,\n        \"num_unique_values\": 0,\n        \"samples\": [],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Model Used\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": null,\n        \"min\": null,\n        \"max\": null,\n        \"num_unique_values\": 0,\n        \"samples\": [],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"A2C Sharpe\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": null,\n        \"min\": null,\n        \"max\": null,\n        \"num_unique_values\": 0,\n        \"samples\": [],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"PPO Sharpe\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": null,\n        \"min\": null,\n        \"max\": null,\n        \"num_unique_values\": 0,\n        \"samples\": [],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"DDPG Sharpe\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": null,\n        \"min\": null,\n        \"max\": null,\n        \"num_unique_values\": 0,\n        \"samples\": [],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"SAC Sharpe\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": null,\n        \"min\": null,\n        \"max\": null,\n        \"num_unique_values\": 0,\n        \"samples\": [],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"TD3 Sharpe\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": null,\n        \"min\": null,\n        \"max\": null,\n        \"num_unique_values\": 0,\n        \"samples\": [],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
            }
          },
          "metadata": {},
          "execution_count": 15
        }
      ],
      "source": [
        "df_summary"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W6vvNSC6h1jZ"
      },
      "source": [
        "<a id='6'></a>\n",
        "# Part 7: Backtest Our Strategy\n",
        "Backtesting plays a key role in evaluating the performance of a trading strategy. Automated backtesting tool is preferred because it reduces the human error. We usually use the Quantopian pyfolio package to backtest our trading strategies. It is easy to use and consists of various individual plots that provide a comprehensive image of the performance of a trading strategy."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "X4JKB--8tj1g"
      },
      "outputs": [],
      "source": [
        "unique_trade_date = processed[(processed.date > TEST_START_DATE)&(processed.date <= TEST_END_DATE)].date.unique()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "q9mKF7GGtj1g",
        "scrolled": true
      },
      "outputs": [],
      "source": [
        "df_trade_date = pd.DataFrame({'datadate':unique_trade_date})\n",
        "\n",
        "df_account_value=pd.DataFrame()\n",
        "for i in range(rebalance_window+validation_window, len(unique_trade_date)+1,rebalance_window):\n",
        "    temp = pd.read_csv('results/account_value_trade_{}_{}.csv'.format('ensemble',i))\n",
        "    df_account_value = df_account_value.append(temp,ignore_index=True)\n",
        "sharpe=(252**0.5)*df_account_value.account_value.pct_change(1).mean()/df_account_value.account_value.pct_change(1).std()\n",
        "print('Sharpe Ratio: ',sharpe)\n",
        "df_account_value=df_account_value.join(df_trade_date[validation_window:].reset_index(drop=True))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oyosyW7_tj1g"
      },
      "outputs": [],
      "source": [
        "df_account_value.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wLsRdw2Ctj1h"
      },
      "outputs": [],
      "source": [
        "%matplotlib inline\n",
        "df_account_value.account_value.plot()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Lr2zX7ZxNyFQ"
      },
      "source": [
        "<a id='6.1'></a>\n",
        "## 7.1 BackTestStats\n",
        "pass in df_account_value, this information is stored in env class\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Nzkr9yv-AdV_",
        "scrolled": true
      },
      "outputs": [],
      "source": [
        "print(\"==============Get Backtest Results===========\")\n",
        "now = datetime.datetime.now().strftime('%Y%m%d-%Hh%M')\n",
        "\n",
        "perf_stats_all = backtest_stats(account_value=df_account_value)\n",
        "perf_stats_all = pd.DataFrame(perf_stats_all)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DiHhM1YkoCel"
      },
      "outputs": [],
      "source": [
        "#baseline stats\n",
        "print(\"==============Get Baseline Stats===========\")\n",
        "df_dji_ = get_baseline(\n",
        "        ticker=\"^DJI\",\n",
        "        start = df_account_value.loc[0,'date'],\n",
        "        end = df_account_value.loc[len(df_account_value)-1,'date'])\n",
        "\n",
        "stats = backtest_stats(df_dji_, value_col_name = 'close')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RhJ9whD75WTs"
      },
      "outputs": [],
      "source": [
        "df_dji = pd.DataFrame()\n",
        "df_dji['date'] = df_account_value['date']\n",
        "df_dji['dji'] = df_dji_['close'] / df_dji_['close'][0] * env_kwargs[\"initial_amount\"]\n",
        "print(\"df_dji: \", df_dji)\n",
        "df_dji.to_csv(\"df_dji.csv\")\n",
        "df_dji = df_dji.set_index(df_dji.columns[0])\n",
        "print(\"df_dji: \", df_dji)\n",
        "df_dji.to_csv(\"df_dji+.csv\")\n",
        "\n",
        "df_account_value.to_csv('df_account_value.csv')\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9U6Suru3h1jc"
      },
      "source": [
        "<a id='6.2'></a>\n",
        "## 7.2 BackTestPlot"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HggausPRoCem"
      },
      "outputs": [],
      "source": [
        "\n",
        "\n",
        "# print(\"==============Compare to DJIA===========\")\n",
        "# %matplotlib inline\n",
        "# # S&P 500: ^GSPC\n",
        "# # Dow Jones Index: ^DJI\n",
        "# # NASDAQ 100: ^NDX\n",
        "# backtest_plot(df_account_value,\n",
        "#               baseline_ticker = '^DJI',\n",
        "#               baseline_start = df_account_value.loc[0,'date'],\n",
        "#               baseline_end = df_account_value.loc[len(df_account_value)-1,'date'])\n",
        "df.to_csv(\"df.csv\")\n",
        "df_result_ensemble = pd.DataFrame({'date': df_account_value['date'], 'ensemble': df_account_value['account_value']})\n",
        "df_result_ensemble = df_result_ensemble.set_index('date')\n",
        "\n",
        "print(\"df_result_ensemble.columns: \", df_result_ensemble.columns)\n",
        "\n",
        "# df_result_ensemble.drop(df_result_ensemble.columns[0], axis = 1)\n",
        "print(\"df_trade_date: \", df_trade_date)\n",
        "# df_result_ensemble['date'] = df_trade_date['datadate']\n",
        "# df_result_ensemble['account_value'] = df_account_value['account_value']\n",
        "df_result_ensemble.to_csv(\"df_result_ensemble.csv\")\n",
        "print(\"df_result_ensemble: \", df_result_ensemble)\n",
        "print(\"==============Compare to DJIA===========\")\n",
        "result = pd.DataFrame()\n",
        "# result = pd.merge(result, df_result_ensemble, left_index=True, right_index=True)\n",
        "# result = pd.merge(result, df_dji, left_index=True, right_index=True)\n",
        "result = pd.merge(df_result_ensemble, df_dji, left_index=True, right_index=True)\n",
        "print(\"result: \", result)\n",
        "result.to_csv(\"result.csv\")\n",
        "result.columns = ['ensemble', 'dji']\n",
        "\n",
        "%matplotlib inline\n",
        "plt.rcParams[\"figure.figsize\"] = (15,5)\n",
        "plt.figure();\n",
        "result.plot();"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oBQx4bVQFi-a"
      },
      "source": []
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.10"
    },
    "pycharm": {
      "stem_cell": {
        "cell_type": "raw",
        "metadata": {
          "collapsed": false
        },
        "source": []
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
